diff options
Diffstat (limited to 'doc/guides/xen/pkt_switch.rst')
| -rw-r--r-- | doc/guides/xen/pkt_switch.rst | 470 |
1 files changed, 470 insertions, 0 deletions
diff --git a/doc/guides/xen/pkt_switch.rst b/doc/guides/xen/pkt_switch.rst new file mode 100644 index 00000000..3a6fc470 --- /dev/null +++ b/doc/guides/xen/pkt_switch.rst @@ -0,0 +1,470 @@ +.. BSD LICENSE + Copyright(c) 2010-2014 Intel Corporation. All rights reserved. + All rights reserved. + + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions + are met: + + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in + the documentation and/or other materials provided with the + distribution. + * Neither the name of Intel Corporation nor the names of its + contributors may be used to endorse or promote products derived + from this software without specific prior written permission. + + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +DPDK Xen Based Packet-Switching Solution +======================================== + +Introduction +------------ + +DPDK provides a para-virtualization packet switching solution, based on the Xen hypervisor's Grant Table, Note 1, +which provides simple and fast packet switching capability between guest domains and host domain based on MAC address or VLAN tag. + +This solution is comprised of two components; +a Poll Mode Driver (PMD) as the front end in the guest domain and a switching back end in the host domain. +XenStore is used to exchange configure information between the PMD front end and switching back end, +including grant reference IDs for shared Virtio RX/TX rings, +MAC address, device state, and so on. XenStore is an information storage space shared between domains, +see further information on XenStore below. + +The front end PMD can be found in the DPDK directory lib/ librte_pmd_xenvirt and back end example in examples/vhost_xen. + +The PMD front end and switching back end use shared Virtio RX/TX rings as para- virtualized interface. +The Virtio ring is created by the front end, and Grant table references for the ring are passed to host. +The switching back end maps those grant table references and creates shared rings in a mapped address space. + +The following diagram describes the functionality of the DPDK Xen Packet- Switching Solution. + + +.. _figure_dpdk_xen_pkt_switch: + +.. figure:: img/dpdk_xen_pkt_switch.* + + Functionality of the DPDK Xen Packet Switching Solution. + + +Note 1 The Xen hypervisor uses a mechanism called a Grant Table to share memory between domains +(`http://wiki.xen.org/wiki/Grant Table <http://wiki.xen.org/wiki/Grant%20Table>`_). + +A diagram of the design is shown below, where "gva" is the Guest Virtual Address, +which is the data pointer of the mbuf, and "hva" is the Host Virtual Address: + + +.. _figure_grant_table: + +.. figure:: img/grant_table.* + + DPDK Xen Layout + + +In this design, a Virtio ring is used as a para-virtualized interface for better performance over a Xen private ring +when packet switching to and from a VM. +The additional performance is gained by avoiding a system call and memory map in each memory copy with a XEN private ring. + +Device Creation +--------------- + +Poll Mode Driver Front End +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Mbuf pool allocation: + + To use a Xen switching solution, the DPDK application should use rte_mempool_gntalloc_create() + to reserve mbuf pools during initialization. + rte_mempool_gntalloc_create() creates a mempool with objects from memory allocated and managed via gntalloc/gntdev. + + The DPDK now supports construction of mempools from allocated virtual memory through the rte_mempool_xmem_create() API. + + This front end constructs mempools based on memory allocated through the xen_gntalloc driver. + rte_mempool_gntalloc_create() allocates Grant pages, maps them to continuous virtual address space, + and calls rte_mempool_xmem_create() to build mempools. + The Grant IDs for all Grant pages are passed to the host through XenStore. + +* Virtio Ring Creation: + + The Virtio queue size is defined as 256 by default in the VQ_DESC_NUM macro. + Using the queue setup function, + Grant pages are allocated based on ring size and are mapped to continuous virtual address space to form the Virtio ring. + Normally, one ring is comprised of several pages. + Their Grant IDs are passed to the host through XenStore. + + There is no requirement that this memory be physically continuous. + +* Interrupt and Kick: + + There are no interrupts in DPDK Xen Switching as both front and back ends work in polling mode. + There is no requirement for notification. + +* Feature Negotiation: + + Currently, feature negotiation through XenStore is not supported. + +* Packet Reception & Transmission: + + With mempools and Virtio rings created, the front end can operate Virtio devices, + as it does in Virtio PMD for KVM Virtio devices with the exception that the host + does not require notifications or deal with interrupts. + +XenStore is a database that stores guest and host information in the form of (key, value) pairs. +The following is an example of the information generated during the startup of the front end PMD in a guest VM (domain ID 1): + +.. code-block:: console + + xenstore -ls /local/domain/1/control/dpdk + 0_mempool_gref="3042,3043,3044,3045" + 0_mempool_va="0x7fcbc6881000" + 0_tx_vring_gref="3049" + 0_rx_vring_gref="3053" + 0_ether_addr="4e:0b:d0:4e:aa:f1" + 0_vring_flag="3054" + ... + +Multiple mempools and multiple Virtios may exist in the guest domain, the first number is the index, starting from zero. + +The idx#_mempool_va stores the guest virtual address for mempool idx#. + +The idx#_ether_adder stores the MAC address of the guest Virtio device. + +For idx#_rx_ring_gref, idx#_tx_ring_gref, and idx#_mempool_gref, the value is a list of Grant references. +Take idx#_mempool_gref node for example, the host maps those Grant references to a continuous virtual address space. +The real Grant reference information is stored in this virtual address space, +where (gref, pfn) pairs follow each other with -1 as the terminator. + + +.. _figure_grant_refs: + +.. figure:: img/grant_refs.* + + Mapping Grant references to a continuous virtual address space + + +After all gref# IDs are retrieved, the host maps them to a continuous virtual address space. +With the guest mempool virtual address, the host establishes 1:1 address mapping. +With multiple guest mempools, the host establishes multiple address translation regions. + +Switching Back End +~~~~~~~~~~~~~~~~~~ + +The switching back end monitors changes in XenStore. +When the back end detects that a new Virtio device has been created in a guest domain, it will: + +#. Retrieve Grant and configuration information from XenStore. + +#. Map and create a Virtio ring. + +#. Map mempools in the host and establish address translation between the guest address and host address. + +#. Select a free VMDQ pool, set its affinity with the Virtio device, and set the MAC/ VLAN filter. + +Packet Reception +~~~~~~~~~~~~~~~~ + +When packets arrive from an external network, the MAC?VLAN filter classifies packets into queues in one VMDQ pool. +As each pool is bonded to a Virtio device in some guest domain, the switching back end will: + +#. Fetch an available entry from the Virtio RX ring. + +#. Get gva, and translate it to hva. + +#. Copy the contents of the packet to the memory buffer pointed to by gva. + +The DPDK application in the guest domain, based on the PMD front end, +is polling the shared Virtio RX ring for available packets and receives them on arrival. + +Packet Transmission +~~~~~~~~~~~~~~~~~~~ + +When a Virtio device in one guest domain is to transmit a packet, +it puts the virtual address of the packet's data area into the shared Virtio TX ring. + +The packet switching back end is continuously polling the Virtio TX ring. +When new packets are available for transmission from a guest, it will: + +#. Fetch an available entry from the Virtio TX ring. + +#. Get gva, and translate it to hva. + +#. Copy the packet from hva to the host mbuf's data area. + +#. Compare the destination MAC address with all the MAC addresses of the Virtio devices it manages. + If a match exists, it directly copies the packet to the matched VIrtio RX ring. + Otherwise, it sends the packet out through hardware. + +.. note:: + + The packet switching back end is for demonstration purposes only. + The user could implement their switching logic based on this example. + In this example, only one physical port on the host is supported. + Multiple segments are not supported. The biggest mbuf supported is 4KB. + When the back end is restarted, all front ends must also be restarted. + +Running the Application +----------------------- + +The following describes the steps required to run the application. + +Validated Environment +~~~~~~~~~~~~~~~~~~~~~ + +Host: + + Xen-hypervisor: 4.2.2 + + Distribution: Fedora release 18 + + Kernel: 3.10.0 + + Xen development package (including Xen, Xen-libs, xen-devel): 4.2.3 + +Guest: + + Distribution: Fedora 16 and 18 + + Kernel: 3.6.11 + +Xen Host Prerequisites +~~~~~~~~~~~~~~~~~~~~~~ + +Note that the following commands might not be the same on different Linux* distributions. + +* Install xen-devel package: + + .. code-block:: console + + yum install xen-devel.x86_64 + +* Start xend if not already started: + + .. code-block:: console + + /etc/init.d/xend start + +* Mount xenfs if not already mounted: + + .. code-block:: console + + mount -t xenfs none /proc/xen + +* Enlarge the limit for xen_gntdev driver: + + .. code-block:: console + + modprobe -r xen_gntdev + modprobe xen_gntdev limit=1000000 + +.. note:: + + The default limit for earlier versions of the xen_gntdev driver is 1024. + That is insufficient to support the mapping of multiple Virtio devices into multiple VMs, + so it is necessary to enlarge the limit by reloading this module. + The default limit of recent versions of xen_gntdev is 1048576. + The rough calculation of this limit is: + + limit=nb_mbuf# * VM#. + + In DPDK examples, nb_mbuf# is normally 8192. + +Building and Running the Switching Backend +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. Edit config/common_linuxapp, and change the default configuration value for the following two items: + + .. code-block:: console + + CONFIG_RTE_LIBRTE_XEN_DOM0=y + CONFIG RTE_LIBRTE_PMD_XENVIRT=n + +#. Build the target: + + .. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + +#. Ensure that RTE_SDK and RTE_TARGET are correctly set. Build the switching example: + + .. code-block:: console + + make -C examples/vhost_xen/ + +#. Load the Xen DPDK memory management module and preallocate memory: + + .. code-block:: console + + insmod ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/xen_dom0/rte_dom0_mm.ko + echo 2048> /sys/kernel/mm/dom0-mm/memsize-mB/memsize + + .. note:: + + On Xen Dom0, there is no hugepage support. + Under Xen Dom0, the DPDK uses a special memory management kernel module + to allocate chunks of physically continuous memory. + Refer to the *DPDK Getting Started Guide* for more information on memory management in the DPDK. + In the above command, 4 GB memory is reserved (2048 of 2 MB pages) for DPDK. + +#. Load uio_pci_generic and bind one Intel NIC controller to it: + + .. code-block:: console + + modprobe uio_pci_generic + python tools/dpdk_nic_bind.py -b uio_pci_generic 0000:09:00:00.0 + + In this case, 0000:09:00.0 is the PCI address for the NIC controller. + +#. Run the switching back end example: + + .. code-block:: console + + examples/vhost_xen/build/vhost-switch -c f -n 3 --xen-dom0 -- -p1 + +.. note:: + + The -xen-dom0 option instructs the DPDK to use the Xen kernel module to allocate memory. + +Other Parameters: + +* -vm2vm + + The vm2vm parameter enables/disables packet switching in software. + Disabling vm2vm implies that on a VM packet transmission will always go to the Ethernet port + and will not be switched to another VM + +* -Stats + + The Stats parameter controls the printing of Virtio-net device statistics. + The parameter specifies the interval (in seconds) at which to print statistics, + an interval of 0 seconds will disable printing statistics. + +Xen PMD Frontend Prerequisites +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. Install xen-devel package for accessing XenStore: + + .. code-block:: console + + yum install xen-devel.x86_64 + +#. Mount xenfs, if it is not already mounted: + + .. code-block:: console + + mount -t xenfs none /proc/xen + +#. Enlarge the default limit for xen_gntalloc driver: + + .. code-block:: console + + modprobe -r xen_gntalloc + modprobe xen_gntalloc limit=6000 + +.. note:: + + Before the Linux kernel version 3.8-rc5, Jan 15th 2013, + a critical defect occurs when a guest is heavily allocating Grant pages. + The Grant driver allocates fewer pages than expected which causes kernel memory corruption. + This happens, for example, when a guest uses the v1 format of a Grant table entry and allocates + more than 8192 Grant pages (this number might be different on different hypervisor versions). + To work around this issue, set the limit for gntalloc driver to 6000. + (The kernel normally allocates hundreds of Grant pages with one Xen front end per virtualized device). + If the kernel allocates a lot of Grant pages, for example, if the user uses multiple net front devices, + it is best to upgrade the Grant alloc driver. + This defect has been fixed in kernel version 3.8-rc5 and later. + +Building and Running the Front End +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +#. Edit config/common_linuxapp, and change the default configuration value: + + .. code-block:: console + + CONFIG_RTE_LIBRTE_XEN_DOM0=n + CONFIG_RTE_LIBRTE_PMD_XENVIRT=y + +#. Build the package: + + .. code-block:: console + + make install T=x86_64-native-linuxapp-gcc + +#. Enable hugepages. Refer to the *DPDK Getting Started Guide* for instructions on + how to use hugepages in the DPDK. + +#. Run TestPMD. Refer to *DPDK TestPMD Application User Guide* for detailed parameter usage. + + .. code-block:: console + + ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" + testpmd>set fwd mac + testpmd>start + + As an example to run two TestPMD instances over 2 Xen Virtio devices: + + .. code-block:: console + + --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" --vdev="eth_xenvirt1;mac=00:00:00:00:00:22" + + +Usage Examples: Injecting a Packet Stream Using a Packet Generator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Loopback Mode +^^^^^^^^^^^^^ + +Run TestPMD in a guest VM: + +.. code-block:: console + + ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" -- -i --eth-peer=0,00:00:00:00:00:22 + testpmd> set fwd mac + testpmd> start + +Example output of the vhost_switch would be: + +.. code-block:: console + + DATA:(0) MAC_ADDRESS 00:00:00:00:00:11 and VLAN_TAG 1000 registered. + +The above message indicates that device 0 has been registered with MAC address 00:00:00:00:00:11 and VLAN tag 1000. +Any packets received on the NIC with these values is placed on the device's receive queue. + +Configure a packet stream in the packet generator, set the destination MAC address to 00:00:00:00:00:11, and VLAN to 1000, +the guest Virtio receives these packets and sends them out with destination MAC address 00:00:00:00:00:22. + +Inter-VM Mode +^^^^^^^^^^^^^ + +Run TestPMD in guest VM1: + +.. code-block:: console + + ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:11" -- -i --eth-peer=0,00:00:00:00:00:22 -- -i + +Run TestPMD in guest VM2: + +.. code-block:: console + + ./x86_64-native-linuxapp-gcc/app/testpmd -c f -n 4 --vdev="eth_xenvirt0,mac=00:00:00:00:00:22" -- -i --eth-peer=0,00:00:00:00:00:33 + +Configure a packet stream in the packet generator, and set the destination MAC address to 00:00:00:00:00:11 and VLAN to 1000. +The packets received in Virtio in guest VM1 will be forwarded to Virtio in guest VM2 and +then sent out through hardware with destination MAC address 00:00:00:00:00:33. + +The packet flow is: + +packet generator->Virtio in guest VM1->switching backend->Virtio in guest VM2->switching backend->wire |