path: root/doc/DMM_DeveloperManual.md

![logo fd.io](resources/logo_fdio-300x184.png)

**DMM Developer Manual**

[**1** **Introduction** 1](#section)

[**2** **DMM Overall Architecture** 2](#dmm-overallarchitecture)

[**3** **Core Components** 3](#core-components)

[**3.1** **nSocket** 3a](#nsocket)

[**3.2** **Framework** 3b](#framework)

[**3.3** **Adaptor** 3c](#adaptor)

[**3.4** **LRD** 3d](#lrd)

[**3.5** **HAL** 3e](#hal)

[**3.6** **OMC** 3f](#omc)

[**4** **Plug-in Architectures** 4](#plug-in-architectures)

[**4.1** **Overview** 4a](#overview)

[**4.2** **Plug-in interface** 4b](#plug-in-interface)

[**4.2.1** **Interface of protocol stack adapter APIs**4ba](#Interface-of-protocol-stack-adapter-APIs)

[**4.2.1.1** **nStack\_module\_info**4baa](#nStack_module_info)

[**4.2.1.2** **ep\_free\_ref**4bab](#ep_free_ref)

[**4.2.1.3** **nstack\_stack\_register\_fn**4bac](#nstack_stack_register_fn)

[**4.2.1.4** **nstack\_proc\_cb**4bad](#nstack_proc_cb)

[**4.2.1.5** **DMM-adapter APIs**4bae](#DMM-adapter-APIs)

[**4.2.1.6** **nstack\_adpt\_init**4baf](#nstack_adpt_init)

[**4.2.2** **epoll architecture** 4bb](#epoll-architecture)

[**4.2.2.1** **ep\_ctl** 4bba](#ep_ctl)

[**4.2.2.2** **ep\_getevt** 4bbb](#ep_getevt)

[**4.2.2.3** **nstack\_event\_callback ** 4bbc](#nstack_event_callback)

[**4.2.2.4** **nsep\_force\_epinfo\_free** 4bbd](#nsep_force_epinfo_free)

[**4.2.3** **select** 4bc](#select)

[**4.2.3.1** **pfselect** 4bca](#pfselect)

[**4.2.4** **fork** 4bd](#fork)

[**4.2.4.1** **fork\_init\_child** 4bda](#fork_init_child)

[**4.2.4.2** **fork\_parent\_fd**4bdb](#fork_parent_fd)

[**4.2.4.3** **fork\_child\_fd** 4bdc](#fork_child_fd)

[**4.2.4.4** **fork\_free\_fd** 4bdd](#fork_free_fd)

[**4.2.5** **Multithreading** 4be](#multithreading)

[**4.2.6** **Resource recovery** 4bf](#resource-recovery)

[**4.2.6.1** **obj\_recycle\_reg** 4bfa](#obj_recycle_reg)

[**4.2.6.2** **obj\_recycle\_fun** 4bfb](#obj_recycle_fun)

[**4.2.7** **LRD**4bg](#LRD)

[**4.2.7.1** **nstack\_rd\_init** 4bga](#nstack_rd_init)

[**4.2.7.2** **nstack\_get\_route\_data** 4bgb](#nstack_get_route_data)

[**4.2.7.3** **nstack\_rd\_get\_stackid** 4bgc](#nstack_rd_get_stackid)

[**5** **Release file** 5](#release-file)

[**5.1** **libs** 5a](#libs)

[**5.2** **Posix API of nSocket** 5b](#posix-api-of-nsocket)

**1. Introduction**
================

This document is used to guide the user-mode protocol stack developers to use
the DMM framework. The document defines the interface that need to be registered
to the DMM when the protocol stack is integrated into the DMM and the interface
that the DMM provides to the protocol stack. In the document as well as in the
code, DMM is also called nStack. The two names are interchangeable.

**2. DMM Overall Architecture**
============================

![nstack architecture](resources/nStack_Architecture.png)

Figure1. The nStack software architecture.HAL: Hardware Abstraction
Layer, and IO adaptor

The DMM framework provides posix socket APIs to the application. A protocol
stack could be plugged into the DMM. DMM will choose the most suitable stack
according to the policy of nRD to application.

nRD is a strategy system which is used to choose a protocol stack for
application based on certain rules, such as network information, SLA, security
and so on.

**3. Core Components**
===================

Figure1 shows an overview of the architecture design for DMM. DMM can be divided
into six main components:  nSocket, Framework (FW), adapters, RD, HAL and OMC
(orchestration\manage\control)..

**3.1 nSocket**
-----------

“nSocket” provides user-friendly POSIX Compatible API of nStack, it intercepts
the socket API calls from the user application via system call hijacking or
the LD_PRELOAD mechanism.

**3.2 Framework**
-------------

Framework is a public framework module that includes shared-memory management,
IPC-management, initialization framework, basic data structures and timer.

**3.3 Adaptor**
-----------

**nStack-adaptor:** Used by protocol stack and implemented by DMM. Provides
interfaces towards DMM.

**Stack-x-adaptor:** Used by DMM and implemented by protocol stack. Provides
interfaces towards protocol stack.

**Kernel-adaptor:** Used and implemented by DMM. Help interact with kernel
stack (if supported).

**3.4 RD**
-------
![RD topology](resources/RD_Topo.PNG)

Resource Discovery subsystem is responsible for “routing/rule” discovery,
negotiation and decision-making (decision to select protocol stack).

**3.5 HAL**
-------

HAL (Hardware Abstraction Layer) layer provides the following functions for
upper-layer protocol stack:

A unified user plane IO interface, shields the lower-level driver implementation
differences. Mask the port implementation differences between IO layer and
Protocol stack.

Initialize the network interface card.

**3.6 OMC**
-------

OMC (operation, Monitor and Control) is responsible for the whole protocol stack
configuration management, monitoring and control.

It provides functionalities like query statistics, monitoring process status etc.

**4 Plug-in Architectures**
=========================

**4.1 Overview**
------------

DMM-Plug-in architecture is an attractive solution for developers seeking to
build protocol stacks that are multi-functional, customizable, and easily
extensible.
![integration scheme](resources/Integration.png)

Figure3. The Integration scheme

Figure 3 shows a typical deployment mode that protocol stack and application
process are deployed in a pipeline mode (DMM also support the run-to-completion
mode) across processes.IPC is used for communication between them.

1. DMM need to integrate the protocol stack adapter-library. The library
   should implement the interfaces defined in the list1. The protocol stack
   adapter is used for communicating with protocol stack process. In addition
   to the function of the protocol stack, this module must implement the
   interfaces defined by DMM, including the socket APIs, the epoll APIs, the
   fork APIs and the resource recycle APIs.

  Interface | detail
  --------- | ------
    ep\_ctl        |   Section 4.2.2.1
    ep\_getevt     |   Section 4.2.2.2
 fork\_init\_child |   Section 4.2.4.1
 fork\_parent\_fd  |   Section 4.2.4.2
 fork\_child\_fd   |   Section 4.2.4.3
 fork\_free\_fd    |   Section 4.2.4.4
 module\_init      |   Section 4.3.5.5

List1. The function provided by protocol adapter.

2. Protocol stack needs to integrate the DMM adapter-library. The library which
   is used to create and initialize shared-memory. The shared-memory is the
   highest performance method of IPC between nSocket and protocol stack.
   The library utilizes the plug-in interface to provide rich features to the
   protocol stack, such as resource recycle and event management.

a.  Protocol stack needs to register the recycle object and functions
    (See section 4.2.7 for details) to the DMM adapter. When the OMC receive
    the process exit signal, DMM adapter processes the exit signal, and
    trigger resource recovery action. Protocol stack does not need to focus
    on the exit of the application.

b.	Protocol stack needs to call the nstack\_adpt\_init
    (See section 4.2.1 for details) defined in the DMM adapter to create and
    initialize the shared memory object used by DMM.The init function
    returns the nStack\_adpt\_cb (See section 4.2.1.1 for details) object
    to the protocol stack. The protocol stack calls the callback functions
    when processing packet.
    Then DMM adapt-library should implement the interface defined in the list2.

  Interface | detail
  --------- | ------
  event\_notify      |  Section 4.2.2.3
  ep\_pdata\_free    |  Section 4.2.2.4
  obj\_recycle\_reg  |  Section 4.2.6.1
  obj\_recycle\_fun  |  Section 4.2.6.2

List2. The function provided by DMM-adaptor. This will be used by protocol stack

c.	In the pipeline mode, initialization of nSocket does not create shared
    memory; instead it looks up and attaches the shared memory created by
    the DMM adapterr.

d.  nRD reads the routing information from the configuration file to make a
    decision as nSocket selects the protocol stack.

With these modules, DMM provides a unified socket interface, and support epoll,
select, fork, zero copy, resource recovery features.

**4.2 Plug-in interface**
---------------------

### **4.2.1 Interface of Stack adapter APIs**

The following APIs must be implemented by the protocol stack adapter.
They are used by nSocket module to do its work.

**4.2.1.1 nStack\_module\_info**

When the protocol stack is integrated into DMM, the first step is adding new
protocol stack module information.


```
typedef struct _ nstack_module_keys {
    const ns_char*  modName;          /*stack name*/
    const ns_char*  registe_fn_name;  /*stack register func name*/
    const ns_char*  libPath;          /*if libtype is dynamic,  it is the path of lib*/
    ns_char         deploytype;       /*depoly model type:  model type1,  model type2,  model type3*/
    ns_char         libtype;          /*dynamic lib or static lib*/
    ns_char         ep_free_ref;      /*when epoll information free,  need to wait that stack would not
                                    notify event */
    ns_char         default_stack;    /*whether is default stack: when don't know how to choose stack,
                                    just use default stack firstly*/
    ns_int32        priority;         /*reserv*/
    ns_int32        maxfdid;          /* the max fd id,  just used to check*/
    ns_int32        minfdid;          /* the min fd id,  just used to check */
    ns_int32        modInx;           /* This is alloced by nStack ,  not from configuration */
} nstack_module_keys;

nstack_module_keys  g_nstack_module_desc[] ={};

typedef enum {
    NSTACK_MODEL_TYPE1,   /*nSocket and stack belong to the same process*/
    NSTACK_MODEL_TYPE2,   /*nSocket and stack belong to different processes,
                           *and nStack don't take care the communication between stack and stack adpt*/
    NSTACK_MODEL_TYPE3,   /*nSocket and stack belong to different processes,  and stax-x-adapter was
                        spplied to communicate whit stack*/
    NSTACK_MODEL_INVALID,
} nstack_model_deploy_type;

```

DMM uses g\_nstack\_module_desc to manage the stack initialization information,
e.g. registration function name, lib path, etc.

When the nSocket calls the nstack\_stack\_registe_fn() (See section 4.2.1.3 for
details) , the information returned by the  protocol stack adapter is stored in
nstack\_module\_info.

```
typedef struct __NSTACK_MODULE {
    char modulename[NSTACK_PLUGIN_NAME_LEN];
    ns_int32  priority;
    void* handle;
    nstack_proc_cb  mops;
    ns_int32        ep_free_ref;  //ep information need free with ref
    ns_int32        modInx;       // The index of module
    ns_int32        maxfdid;      //the max fd id
    ns_int32        minfdid;      //the min fd id
} nstack_module;

typedef struct{
    ns_int32 modNum;        // Number of modules registed
    ns_int32 linuxmid;
    ns_int32 spmid;         /* Stackx module index. */
    nstack_module *defMod;  // The default module
    nstack_module modules[NSTACK_MAX_MODULE_NUM];
} nstack_module_info;

nstack_module_info g_nstack_modules = {
    .modNum = 0,
    .linuxmid = -1,
    .modules = {{{0}}},
    .defMod = NULL,
}

```

CS2. The information of nstack\_module\_info

 **4.2.1.2 ep\_free\_ref**

This field in data structure nstack\_module\_keys is used in pipe-line mode.
When the application process exits, the socket resources in the application
process are released, but the resources in the protocol stack are not released.
At this time, the protocol stack needs to continue accessing the epoll shared
resource to send events.

So, you cannot free the resources of epoll immediately after the application
exits. Only when the resources of the protocol stack are completely released,
the epoll share resource can be released and it is done by the interface
provided by the DMM adapter.

This field marks whether the stack needs to release the epoll resources. If it
is 1, DMM will not release resources when close is called, and the stack must
calls the interface provided by DMM adapter to release after releasing its own
resources. If it is 0, DMM(nSocket) released the source directly.

**4.2.1.3 nstack\_stack\_register\_fn**

Function prototypes:

  `typedef int (*nstack_stack_register_fn) (nstack_proc_cb *proc_fun, nstack_event_cb *event_ops);`


CS3. The declaration of nstack\_stack\_register\_fn

There are three parameters in this function.

nstack\_socket\_ops: defined in declare\_syscalls.h.tmpl file and is
same is in Posix.

nstack\_event\_ops:

```
typedef struct __nstack_event_cb
{
  void *handle;			/*current so file handler */
  int type;			/*nstack is assigned to the protocol stack and needs to be passed to nstack when the event is reported */
  int (*event_cb) (void *pdata, int events);
} nstack_event_cb;

```

Detail in the Section 4.2.3

**4.2.1.4 nstack\_proc\_cb**

```
typedef struct __nstack_proc_ops{
    nstack_socket_ops socket_ops; /*posix socket api*/
    nstack_extern_ops extern_ops; /*other proc callback*/
} nstack_proc_cb;

typedef struct __nstack_extern_ops {
  int (*module_init) (void);    /*stack module init */
  int (*fork_init_child) (pid_t p, pid_t c);    /*after fork, stack child process init again if needed. */
  void (*fork_parent_fd) (int s, pid_t p);      /*after fork, stack parent process proc again if needed. */
  void (*fork_child_fd) (int s, pid_t p, pid_t c);      /*after fork, child record pid for recycle if needed. */
  void (*fork_free_fd) (int s, pid_t p, pid_t c);       /*for SOCK_CLOEXEC when fork if needed. */
  unsigned int (*ep_ctl) (int epFD, int proFD, int ctl_ops, struct epoll_event * event, void *pdata);   /*when fd add to epoll fd, triggle stack to proc if need */
  unsigned int (*ep_getevt) (int epFD, int profd, unsigned int events); /*check whether some events exist really */
  int (*ep_prewait_proc) (int epfd);               /* The pretreatment before epwait by stack-x */
  int (*stack_fd_check) (int s, int flag);      /*check whether fd belong to stack, if belong, return 1, else return 0 */
  int (*stack_alloc_fd) ();     /*alloc a fd id for epoll */
  int (*peak) (int s);          /*used for stack-x , isource maybe no need */
} nstack_extern_ops;

```

### **4.2.1.5 DMM-adapter APIs**

For the multi-process model, DMM provide a library with resource initialization,
resource recycling, event notification and other functions to the protocol stack.

nstack_adpt_fun is the core object between protocol stack and DMM.
It defines some pointers which can be used by protocol stacks.

```
typedef struct nsfw_com_attr
{
  int policy;
  int pri;
} nsfw_com_attr;

typedef struct __nstack_dmm_para
{
  nstack_model_deploy_type deploy_type;
  int proc_type;
  nsfw_com_attr attr;
  int argc;
  char **argv;
} nstack_dmm_para;

int nstack_adpt_init (nstack_dmm_para * para)

```

### **4.2.1.5 nstack\_adpt\_init**

  `int nstack_adpt_init (nstack_dmm_para * para)`


**Description**:

The protocol stack calls this function to initialize the DMM framework and get
the callback function from DMM.

**Parameters:**

*deploy\_type*: decides whether protocol stack is deployed in same process along
with DMM or in a separate process.
*flag*: reserved
*out\_ops*: DMM provides the interfaces to protocol Stack, defined in
nstack\_adpt\_fun(details in 4.2.1)
*in\_ops*: protocol stack callback function registered to DMM.

### **4.2.2 epoll Architecture**

The function of these data structures is similar to the one in linux.
```
struct eventpoll {
    sys_sem_st lock;        /*eventpoll lock*/
    sys_sem_st sem;         /* rdlist lock */
    sem_t waitSem;          /*sem for epoll_wait */
    struct ep_hlist rdlist; /*ready epitem list*/
    struct ep_hlist txlist; /*pad*/
    struct ep_rb_root rbr;  /* rbtree*/
    int epfd;               /*epoll  fd*/
    u32 pid;              /* Pid of the process who creat the structure,  the resources used to release */
    nsfw_res res_chk;       /* Verify whether resources are repeatedly released */
};
```

```
typedef struct{
    int iindex;
    int iNext;
    int fd;
    i32 fdtype;     /* 0:  socket fd,  1:  epoll fd */
    i32 rlfd;       /* copy of fdInf->rlfd */
    i32 rmidx;      /* copy of fdInf->rmidx */
    i32 protoFD[NSEP_SMOD_MAX];
    i32 epaddflag[NSEP_SMOD_MAX];
    struct eventpoll *ep;
    sys_sem_st epiLock;
    sys_sem_st freeLock;
    struct ep_list epiList;    /* This restore the epitem of this file descriptor */
    u32 sleepTime;   /* add for NSTACK_SEM_SLEEP */
    nsep_pidinfo pidinfo;
    nsfw_res res_chk;
    void* private_data;
    i32 reserv[4];
} nsep_epollInfo_t;

```

```
struct epitem {
    struct ep_rb_node rbn;
    struct ep_hlist_node rdllink;
    struct ep_hlist_node lkFDllink;
    int nwait;
    struct eventpoll *ep;
    nsep_epollInfo_t *epInfo;
    struct epoll_event event;
    struct list_node fllink;
    struct ep_hlist_node txlink;
    unsigned int revents;
    int fd;
    u32 pid;
    void* private_data;
    nsfw_res res_chk;
};

```

When the application calls epoll\_create(), nSocket first creates socket or
epoll object in the data type of nsep\_epollInfo\_t, then creates an eventpoll
object. For the created nsep_epollInfo_t object, when the file descriptor is an
epoll socket, ep points to eventpoll and the eplist is not used. When the file
descriptor is a normal socket, the epiList restores the fd and the ep is not
used.

When the application calls epoll\_ctl() to add a socket to the epoll, nSocket
allocates an epitem object. The epitem object is added to a rbtree pointed by
epList in nsep\_epollInfo\_t, which is the epoll created in the previous step.
In the epitem object just allocated, nSocket uses ep to point to eventpoll,
revents to store event that is produced by the protocol stack,  epinfo to point+
to the epoll object in the data type of nsep\_epollInfo\_t. This is how DMM
connects the epoll, event, and socket together.

When the application calls epoll\_wait(), nSocket scans the rdlist in eventpoll
to get events and return them to the application.

Figure 3 is Runtime Graphs of nSocket in epoll function:

![ePoll flow chart](resources/Epoll_flwChart.png)

Figure 4: flow chart of various epoll functions function:

![event and ep_pData free flow](resources/EvntNotify_StackX.png)

Figure5: flow chart of nstack_event_callback and nsep_force_epinfo_free in protocol Stack

### **4.2.2.1 ep\_ctl**

 `int (*ep_ctl)(int proFD,  int ctl_ops,  struct epoll_event *event,  void *pdata);`


**Description:**

1\. Used by the application to inform protocol stack to add/modify/delete file
descriptor (FD) to/from epoll and monitor events.

2\. When the application adds new events or modifies existing events, it gets
prompt notification。

3\. pdata stores the nsep_epollInfo_t corresponding to the FD. It is used to
pass the value to the protocol stack event notification API when an event occurs.

**Parameters:**

*proFD*: sock file descriptor created by the protocol stack.
*ctl\_ops*: 0 add, 1 modify, 2 delete。
*event*: similar to Linux “struct epoll\_event”。
*pdata*: Private data that DMM passes to the protocol stack.

**Interface implemented by:** Protocol stack.

### **4.2.2.2 ep\_getevt**

**Interface Definition:**

`void (*ep_getevt)(int epFD,  int profd,  unsigned int events);`

**Description:**

To check if an event in protocol stack exist. Due to multi-threading or other
reasons, it’s possible that protocol stack is busy and causes delay in event
notification. Or event may be cleared by some application operation. Application
may use the API to check whether the event still exists.

**Parameters:**

*epFD*: epoll FD.
*proFD*: sock file descriptor created by the protocol stack.
*event*: Events which need to check.

**Interface implemented by:** Protocol stack.

### **4.2.2.3 nstack\_event\_callback **

`int (*nstack_event_callback)(void *pdata, int events)`

**Description:**

Called by the protocol stack when there is an event occurs.

**Parameters:**

*pdata*: pdata is the private data passed in ep\_ctl()..
*events*: Whatever event occurred

**Interface implemented by:** DMM

### **4.2.2.4 nsep\_force\_epinfo\_free**

`int (*nsep_force_epinfo_free)(void *pdata)`

**Description:**

When closes fd, the protocol stack frees the resources related to fd. This API
is called by the protocol stack to release private data.

**Precautions for use:**

This API should only be called when ep\_free\_ref is configured 1 in
nstack\_module\_keys.

**Parameters:**

*pdata*: pdata is the private data in ep\_ctl().

**Interface implemented by:** DMM

### **4.2.3 select**

Select requires the protocol stack adapter module to provide a select API in
order to query for existing events. When the application calls select(),
DMM starts a separate thread to fetch the select event for each stack and
sends it back to the application.

![select flowchart](resources/SelectFunc.png)

Figure 6: flow chart of select function

### **4.2.3.1 pfselect**

`int *pfselect(int s, fd_set* readfds, fd_set* writefds, fd_set* exceptfds, struct timeval* timeout)`


**Description:**

Query whether there is a corresponding protocol stack event.

**Parameters:**

*s*: The maximum file descriptor value in the collection
*readfds*: file descriptor set will be watched for read event
*writefds*: file descriptor set will be watched for write event
*exceptfds*: file descriptor set will be watched for exception cases

**Interface implemented by:** Protocol Stack

### **4.2.4 fork**

fork() creates a child process by making an exact duplicate of the calling
process in a separate address spaces. Each process, parent and child, has its
own process address space where memory segments, such as code segment, data
segment, stack segment, etc., are placed.

There are few exceptions to this.

1.  The fork system call duplicates all the file descriptors of the parent into
    the child, so that the file descriptors in the parent and child point to the
    same files.

2. 	The shared memory is still shared between parent and child process after
    fork ().

3.  Child process clones only thread, which executed it.

When parent process or child process releases the shared resources on process
quit or kill, it affects the other process.

Also if the child process wants to inherit other threads in the parent process,
the child process needs to pull or create those threads after the fork.

Both parent and child process can access the shared memory, it create conflicts.

Because of these issues, DMM provides the fork APIs to take care of processing
before and after calling the Linux fork interface. It includes interface lock
and shared resource lock, reference count incrimination, saving the PID of the
subprocesses, reinitialization and SOCK\_CLOEXEC option in socket special
handlings.

Each shared memory data structure in DMM contains a reference count and a list
of process pids, so that the shared memory can be managed between parent and
child processes. On each fork, the reference count is increased by 1 and the
child process id is added to the pid array.

```
struct  xxx{
    xxx，
    int ref； / * Reference count * /
    pid_t  pid_array[];
        / * for father and son after the process list * /
};

```

DMM needs assistance from protocol stack to release or hold the shared
resources. Protocol stack is to provide stack specific implementation on
these DMM APIs.

The following flow chart shows how fork process is handled:

![fork process handling](resources/ForkChild.png)

#### **4.2.4.1 fork\_init\_child**

`int (* fork_init_child) (pid_t p, pid_t c)`

**Description:**

After fork, the API is used to add child process pid into the pid list in the
share socket data structure. When the child process exits, only the resources
in the child process are released unless the shared resource reference count
reaches 0.

**Parameters:**

*p*: parent process pid

*c*: child process pid


**Interface Implemented by:** Protocol stack.

#### **4.2.4.2 fork\_parent\_fd**

`void (* fork_parent_fd)(int s, pid_t p)`

**Description:**

For the socket with the SOCK_CLOEXEC flag on, when it is closed after fork,
only the file descriptor is recycled, no other resource is released.
It is a special treatment for SOCK_CLOEXEC.

**Parameters:**

*s*: socket fd

*p*: parent process pid

**Interface Implemented by:** Protocol stack.

#### **4.2.4.3 fork\_child\_fd**

`void (* fork_child_fd) (int s, pid_t p, pid_t c);`

**Description:**

After fork, the API is used to add child process pid into the pid list in the
share socket data structure. When the child process exits, only the resources
in the child process are released unless the shared resource reference count
reaches 0.

**Parameter Description:**

*s*: file descriptor of the socket to be operated

*p*: parent process pid

*c*: child process pid

**Interface Implemented by:** Protocol stack.

####  **4.2.4.4 fork\_free\_fd**

`void (* fork_free_fd) (int s, pid_t p, pid_t c);`

**Description:**

For the socket with the SOCK\_CLOEXEC flag on, when it is closed after fork,
only the file descriptor is recycled, no other resource is released.
It is a special treatment for SOCK\_CLOEXEC.

**Parameters:**

*s*: socket fd.

*p*: parent process pid.

*c*: child process pid.

**Interface Implemented by:** Protocol stack

### **4.2.5 Multithreading**

nSocket supports multi-thread. nSocket uses a ref and state to support
the feature.

### **4.2.6 Resource recovery**

Protocol stack must implement resource recovery. When the application crashes
or exits, the resources allocated by protocol stack must be released,
otherwise it causes resources leaking and can crash the system.

DMM provides the following mechanism to support the feature.

There is certain relationship between resource recovery and fork. Each fork
increases the reference count by 1 and saves the child process PID to the
corresponding shared resource. On exiting of the process, decreases the
reference count by 1 if the exiting process PID equals to one of the saved
PIDs and releases the shared resource if the reference count is 0, do not
release the shared resource if the reference count in not zero.

1.  There are two kinds of resource recovery, normal exit and abnormal
    process exit. Resources are freed on normal exit operation. But on the
    abnormal process exits, there is no one to clean up the shared resources
    or decrease the reference count, so we rely on the separate process to
    monitor the processes and notify the stack process to release the resources.

2.  To support resource recovery, the protocol stack needs to define a resource
    object type in responding to DMM request and registers the recycle function
    at initialization time.

#### **4.2.6.1 obj\_recycle\_reg**

TO Do

#### **4.2.6.2 obj\_recycle\_fun**

TO DO

### **4.2.7 LRD**

LRD function is mainly the protocol stack routing function, the internal storage
of a protocol stack selection policy table. When multiple protocol stacks are
integrated, according to the incoming IP address, protocol stack type and other
information, you can select a suitable protocol stack from the protocol Stack
selection policy table. During nSocket module initialization, it invokes the
LRD module initialization API to register the protocol stack information and
the protocol stack selection policy gets the interface.

The initialization function for the LRD module is defined as follows.



```
int nstack_rd_init (nstack_stack_info *pstack, int num, nstack_get_route_data *pfun, int fun_num)

typedef struct __nstack_rd_stack_info
{
    /*stack name */
    rd_stack_plane_map stack;
    /*stack id */
    int stack_id;
    /*when route info not found, high priority stack was chose, same priority chose fist input one */
    int priority;                 /*0: highest: route info not found choose first */
} nstack_rd_stack_info; 

/*rd local data*/
typedef struct __rd_local_data {
    nstack_rd_stack_info *pstack_info;
    int stack_num;
    nstack_rd_list route_list[RD_DATA_TYPE_MAX];  /*route table */
    nstack_get_route_data sys_fun[NSTACK_SYS_FUN_MAX];    /*rd data sys proc function list */
    int fun_num;
} rd_local_data;

rd_local_data *g_rd_local_data = NULL;

/*get rd info. if return ok, data callee alloc memory, caller free, else caller don't free*/
int (*nstack_get_route_data)(rd_route_data **data, int *num);

```

When a new protocol stack is integrated into DMM, the protocol stack selection
configuration file will not directly match with the protocol stack name, but
with the plane name, which is more descriptive about stack and other interface
technology, therefore the RD has a map between the protocols stack and plane
name.

```
typedef struct __rd_stack_plane_map{
    char stackname[STACK_NAME_MAX];   /* Stack name */
    char planename[RD_PLANE_NAMELEN];   /* Plane name */
    int stackid;       /*stack id */
} rd_stack_plane_map;

rd_stack_plane_map g_nstack_plane_info[] = {
   {{“stackx”},  {“nstack-dpdk”} , -1},
   {{“kernel”},  {“nstack-linux”} , -1},
};

```

When application calls the DMM socket, connect, sendto, sendmsg APIS, it
triggers the nSocket module to invoke the nstack\_rd\_get\_stackid() API to get
the route information. The protocol stack is then selected based on the
returned protocol stack ID.

#### **4.2.7.1 nstack\_rd\_init**

`int nstack_rd_init (nstack_stack_info * pstack, int num, nstack_get_route_data *pfun, int fun_num)`

**Description:**

Initializes the RD module g\_rd\_local\_data to receive and save the protocol
stack information provided by nSocket. Also provides the API to get protocol
stack selection information.

When initialized, nSocket passes all the integrated protocol stacks information
to the RD module. The information includes the protocol stack name, nSocket
assigned stack id after integration and selects priority

**Parameters:**

*pStack*: a list of the protocol stack info
*num*: the number of protocol stack info passed in.
*pfun*: function to get the protocol stack selection information.
*fun_num*: the number of pfun passed in

**Interface implemented by:**DMM

#### **4.2.7.2 nstack\_get\_route\_data**

`typedef int (*nstack_get_route_data) (rd_route_data ** data, int *num);`

**Description:**

Obtain the protocol stack selection information according to the protocol
stack plane name..

**Parameters:**

*planename*: description on stack and other interface technology. nSocket
provides the protocol stack name. The configuration file saves an
outbound interface type, which is corresponding to a protocol stack.
*data*: protocol stack selection information.

```
typedef enum __rd_data_type{
   RD_DATA_TYPE_IP,       / * According to the ip address to select the protocol stack * /
   RD_DATA_TYPE_PROTO,    / * Select protocol stack by protocol type * /
   RD_DATA_TYPE_MAX,
}rd_data_type;

typedef struct __rd_route_data{
    rd_data_type type; / * Stored protocol stack selection type * /
    char stack_name[RD_PLANE_NAMELEN];    / * Stack plane name */
    union {
       rd_ip_data ipdata;        / * Select the IP address of the protocol stack type * /
       unsigned int proto_type;  / * Select protocol information for this stack * /
    };
} rd_route_data;

```

*num*: The number of routing information returned.

**Interface implemented by:**DMM

#### **4.2.7.3 nstack\_rd\_get\_stackid**

`int nstack_rd_get_stackid(nstack_rd_key* pkey, int *stackid) ;`

**Description:**

Select parameters according to the protocol stack to obtain the protocol stack
id corresponding to the parameter.

**Parameter:**

*pkey*: protocol stack selection key.

```
typedef struct __nstack_rd_key{
    rd_data_type type;       / * Select protocol type parameters * /
    union {
        unsigned int ip_addr;  / * This field is valid if the type is RD_DATA_TYPE_IP, which is
                                   the IP address of the connection to be initiated * /
        unsigned int proto_type; / * This field is valid if the type is RD_DATA_TYPE_PROTO, which
                                   is the protocol type of the socket created at the time * /
    };
} nstack_rd_key;

```

*stackid*: output parameter; nSocket internal number of the protocol stack id.

**Interface implemented by:** DMM

**5 Release file**
================

**5.1 libs**
--------

**libnStackAPI.so:** The API library that DMM provides to the application.
--------------------------------------------------------------------------------

**libnStackAdapt.so:** DMM provides an adapter library for cross process
protocol stack connections.

**5.2 Posix API of nSocket**
------------------------

DMM provides socket APIs and integrates various protocol stacks. DMM selects
and invokes a specific protocol stack based on the information, such as IP and
protocol type, from the application. The protocol stacks do not interact with
the application. After the selection, what features an API supports are
determined by the implementation provided by the protocol stack itself.

The standard socket APIs defined by DMM are as follows:

  Interface definition                                                      |            Interface Description
  --------------------                                                      |            ---------------------
  int socket(int, int, int)                                                     |            same as corresponding POSIX apis
  int bind(int, const struct sockaddr\*, socklen\_t)                            |            same as corresponding POSIX apis
  int listen(int, int)                                                          |            same as corresponding POSIX apis
  int shutdown(int, int)                                                        |            same as corresponding POSIX apis
  int getsockname(int, struct sockaddr\*, socklen\_t\*)                         |            same as corresponding POSIX apis
  int getpeername(int, struct sockaddr\*, socklen\_t\*)                         |            same as corresponding POSIX apis
  int getsockopt(int, int, int, void\*, socklen\_t\*)                           |            same as corresponding POSIX apis
  int setsockopt(int, int, int, const void\*, socklen\_t)                       |            same as corresponding POSIX apis
  int accept(int, struct sockaddr\*, socklen\_t\*)                              |            same as corresponding POSIX apis
  int accept4(int, struct sockaddr\*, socklen\_t\*, int flags)                  |            same as corresponding POSIX apis
  int connect(int, const struct sockaddr\*, socklen\_t)                         |            same as corresponding POSIX apis
  ssize\_t recv(int , void\*, size\_t, int)                                     |            same as corresponding POSIX apis
  ssize\_t send(int, const void\*, size\_t, int)                                |            same as corresponding POSIX apis
  ssize\_t read(int, void\*, size\_t)                                           |            same as corresponding POSIX apis
  ssize\_t write(int, const void\*, size\_t)                                    |            same as corresponding POSIX apis
  ssize\_t writev(int, const struct iovec \*, int)                              |            same as corresponding POSIX apis
  ssize\_t readv(int, const struct iovec \*, int)                               |            same as corresponding POSIX apis
  ssize\_t sendto(int, const void \*, size\_t, int, const struct sockaddr \*, socklen\_t) |  same as corresponding POSIX apis
  ssize\_t recvfrom(int, void \*, size\_t, int, struct sockaddr \*, socklen\_t \*)  |        same as corresponding POSIX apis
  ssize\_t sendmsg(int, const struct msghdr \*, int flags)                          |        same as corresponding POSIX apis
  ssize\_t recvmsg(int, struct msghdr \*, int flags)                                |        same as corresponding POSIX apis
  int close(int)                                                                    |        same as corresponding POSIX apis
  int select(int, fd\_set\*, fd\_set\*, fd\_set\*, struct timeval\*)                |        same as corresponding POSIX apis
  int ioctl(int, unsigned long, unsigned long)                                      |        same as corresponding POSIX apis
  int fcntl(int, int, unsigned long)                                                |        same as corresponding POSIX apis
  int epoll\_create(int)                                                            |        same as corresponding POSIX apis
  int epoll\_ctl(int, int, int, struct epoll\_event \*)                             |        same as corresponding POSIX apis
  int epoll\_wait(int, struct epoll\_event \*, int, int)                            |        same as corresponding POSIX apis
  pid\_t fork(void)                                                                 |        Before/after calling base fork() need to handle other processing like ref\_count etc.