path: root/trex_book.asciidoc

TRex 
====
:author: hhaim 
:email: <hhaim@cisco.com> 
:revnumber: 1.77-0.0
:quotes.++:
:numbered:
:web_server_url: http://trex-tgn.cisco.com/trex
:local_web_server_url: csi-wiki-01:8181/trex


== Change log

[options="header",cols="^1,^h,3a"]
|=================
| Version |  name   |  meaning
| 1.77-0.0    |  Hanoh Haim (hhaim) |
- initail versions
| 1.77.1    | Dan Klein (danklei)
|
- fixed ISO images references and supported linux supported OS

|=================


== Introduction

=== A word on traffic generators

Traditionally, routers have been tested using commercial traffic generators, while performance
typically has been measured using packets per second (PPS) metrics. As router functionality and
services have become more complex, stateful traffic generators have become necessary to
provide more realistic application traffic scenarios.
The advantages of realistic traffic generators are:

* Providing more accurate performance numbers
* Finding real bottlenecks

==== Current Challenges:

* *Cost* : Commercial State-full traffic generators are expensive
* *Scale* : Bandwidth does not scale up well with features complexity
* *Standardization* : Lack of standardization of traffic patterns and methodologies
* *Flexibility* : Commercial tools do not allow agility when flexibility and changes are needed

==== Implications 

* High capital expenditure (capex) spent by different teams
* Testing in low scale and extrapolation became a common practice, it is not accurate, and hides real life bottlenecks and quality issues
* Different feature / platform teams benchmark and results methodology
* Delays in development and testing due to testing tools features dependency
* Resource and effort investment in developing different ad hoc tools and test methodologies

=== Overview of TRex

TRex addresses these problems through an innovative and extendable software implementation and by leveraging standard and open SW and x86/UCS HW.

* Generates and analyzes L4-7 traffic and able to provide in one tool capabilities provided by commercial L7 tools.
* Stateful traffic generator based on pre-processing and smart replay of real traffic templates.
* Generates and *amplifies* both client and server side traffic.
* Customized functionality can be added.
* Scale to 200Gb/sec for one UCS ( using Intel 40Gb/sec NICS)
* Low cost
* Virtual interfaces support, enable TRex to be used in a fully virtual environment without physical NICs and the following example use cases:
** Amazon AWS
** Cisco LaaS
** TRex on your laptop
** Self-contained packaging that can be easily installed and deployed


.TRex Hardware 
[options="header",cols="1^,1^"]
|=================
|Cisco UCS Platform | Intel NIC 
| image:images/ucs200_2.png[title="generator"] | image:images/Intel520.png[title="generator"]  
|=================

=== Purpose of this guide

This guide explains the use of TRex internals and the use of TRex in conjunction with Cisco ASR1000 Series routers. The examples illustrate novel traffic generation techniques made possible by TRex. 

== Download and installation

=== Hardware recommendation

TRex operates in a Linux application environment, interacting with Linux kernel modules. 
TRex curretly works on x86 architecture and can operates well on Cisco UCS hardware. The following platforms have been tested and are recommended for operating TRex. 

[NOTE]
=====================================
 A high-end UCS platform is not required for operating TRex in its current version, but may be required for future versions.
=====================================
 

.Preferred UCS 
[options="header",cols="1,3"]
|=================
| UCS Type | Comments 
| UCS C220 M3/M4  | Supports up to 40Gb/sec with 540-D2 and with newer Intel NIC 80Gb/sec with 1RU, recommended
| UCS C200| Early UCS model
| UCS C210 M2 | Supports up to 40Gb/sec  PCIe3.0
| UCS C240 M3 | Supports up to 200Gb/sec  using Intel XL710 NICS
| UCS C260M2 | Supports up to 30Gb/sec due to V2 PCIe. 
|=================

.Internal Components
[options="header",cols="1,2",width="60%"]
|=================
| Components |  Details 
| CPU  | 2x CPU E5-2620 
| CPU Configuration | 2-Socket CPU configurations (can also work with one CPU)	 
| Memory | 2x4 banks for each CPU. Total of 8 BANKS ==> 32GB	 
| NO RAID | NO RAID
|=================
 
.Intel NICS supported
[options="header",cols="1,1,2",width="50%"]
|=================
| Bandwidth | Chipset |  Example
| 1Gb/sec  | Intel I350 | Intel 4x1GE 350-T4 NIC
| 10Gb/sec | Intel 82599  | Intel x520-D2  Cisco Order tool 2X Intel N2XX-AIPCI01, Intel X520 Dual Port 10Gb SFP+ Adapter
| 40Gb/sec | Intel XL710  | QSFP+
| VMXNET   | VMware paravirtualize  | connect using vmWare vSwitch
| E1000    | paravirtualize  | vmWare/KVM/VirtualBox 
|=================

IMPORTANT: Intel SFP+ 10Gb/Sec is the only one supported by default on the standard Linux driver. TRex also supports Cisco 10Gb/sec SFP+. 

.Sample order for UCSC-C220-M3S with 4x10Gb ports
[options="header",cols="2,1^",width="50%"]
|=================
| Component  | Amount
| UCSC-C220-M3S     |  1
| UCS-CPU-E5-2650   |  2
|  UCS-MR-1X041RY-A |  8
|  A03-D500GC3      |  1
|  N2XX-AIPCI01     |  2
|  UCSC-PSU-650W    |  1
|  SFS-250V-10A-IS  |  1
|  UCSC-CMA1        |  1
|  UCSC-HS-C220M3   |  2
|  N20-BBLKD        |  7
|  UCSC-PSU-BLKP    |  1
|  UCSC-RAIL1       |  1
|========================  

NOTE: You should buy seperatly the 10Gb/sec SFP+, Cisco would be fine with TRex ( but not for plain Linux driver ).

=== Install OS 

==== Supported versions

Fedora 18-20 , and Ubuntu 14.04.1 LTS are the Linux OS supported. 
You should install the *64bit* Kernel version.
More 64bit OS could be supported by compiling the drivers.

WARNING: Only *64bit* Kernels are supported 

To verify that your kernel is 64bit version try this

[source,bash]
----
$uname -m
x86_64   #<1>
----
<1> x86_64 is the desired output 


==== Download ISO file 

The ISO images of the described Linux OS can be downloaded from the following links:

.Supported Linux ISO image links
[options="header",cols="1,2,3^",width="50%"]
|======================================
| # | Distribution | SHA256 Checksum
| 1.| link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/18/Fedora/x86_64/iso/Fedora-18-x86_64-DVD.iso[Fedora 18] 
    | link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/18/Fedora/x86_64/iso/Fedora-18-x86_64-CHECKSUM[Fedora 18 CHECKSUM] 
| 2.| link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/19/Fedora/x86_64/iso/Fedora-19-x86_64-DVD.iso[Fedora 19]
    | link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/19/Fedora/x86_64/iso/Fedora-19-x86_64-CHECKSUM[Fedora 19 CHECKSUM] 
| 3.| link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/19/Fedora/x86_64/iso/Fedora-20-x86_64-DVD.iso[Fedora 20]
    | link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-CHECKSUM[Fedora 20 CHECKSUM] 
| 4.|link:http://old-releases.ubuntu.com/releases/14.04.1/ubuntu-14.04-desktop-amd64.iso[Ubuntu 14.04.01]
    | http://old-releases.ubuntu.com/releases/14.04.1/SHA256SUMS[Ubuntu 14.04 CHECKSUM]
|======================================


Then, verify the checksum of the downloaded file matches the linked checksum values with the `sha256sum` command. For example:

[source,bash]
----
$sha256sum Fedora-18-x86_64-DVD.iso
91c5f0aca391acf76a047e284144f90d66d3d5f5dcd26b01f368a43236832c03 #<1>
----
<1> Should be equal to the sha256 values described in the linked CHECKSUM files. 

==== Install Linux 

Ask your lab admin to install the Linux using CIMC, assign an IP, and set the DNS. Request the sudo or super user password to enable you to ping and SSH.

IMPORTANT: To use TRex, you should have sudo on this machine or root password.
WARNING: Upgrading the linux Kernel using `yum upgrade` require to build the TRex drivers. 

==== Verify Intel NIC installation

The following is an example of 4x10Gb/sec TRex with I350 management port and four x520-D2 (82599 chipset):

[source,bash]
----
$[root@trex]lspci | grep Network
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<1>
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<2>
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) #<3>
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
82:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
82:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
----
<1> Management port 
<2> CIMC port
<3> 10Gb/sec traffic ports ( Intel 82599EB) 

=== Obtaining the TRex package

Connect by ssh to the TRex machine and do the following:

assuming *$WEB_URL* is *{web_server_url}* or *{local_web_server_url}* (cisco internal)

[source,bash]
----
$mkdir trex
$cd trex
$wget --no-cache $WEB_URL/release/latest 
$tar -xzvf latest 
----


to take the bleeding edge version 
[source,bash]
----
$wget --no-cache $WEB_URL/release/be_latest 
----

To obtain a specific version, do the following: 
[source,bash]
----
$wget --no-cache $WEB_URL/release/vX.XX.tar.gz #<1>
----

=== Running TRex for the first time in loopback 

If you have 10Gb/sec TRex (based on Intel 520-D2 NICs) you can verify that it works correctly by loopback the ports.
You can install Intel SFP+ or Cisco SFP+, but you cannot connect ports that are on the same NIC to each other (it might not sync).
If you have only one NIC of 10gb/sec you cannot perform this test beacause the ports will not have valid link. 
Another option for loopback is to use Cisco twinax copper cable see link:http://www.fiberopticshare.com/tag/cisco-10g-twinax[here]

//TBD: perhaps rephase, using a "Prerequisites" or "Required" heading. The requirement here would be: Two (2) 10gb/sec NICs 
//[hh] it is not accurate beacuse with 1Gb/sec you can have this test 

.Correct loopback
image:images/loopback_right.png[title="rigt"] 

.Wrong loopback
image:images/loopback_wrong.png[title="rigt"] 

In case you have 1Gb/Sec Intel NIC (I350) you can do anything you like from the loopback perspective *but* you must filter the management port before see xref:trex_config[here].

==== Identify the ports 

[source,bash]
----
 $>sudo ./dpdk_setup_ports.py --s

 Network devices using DPDK-compatible driver
 ============================================

 Network devices using kernel driver
 ===================================
 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active*
 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<1>
 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<2>
 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<3>
 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<4>


 Other network devices
 =====================
 <none>
----
<1> TRex interface #1 before unbinding 
<2> TRex interface #2 before unbinding 
<3> TRex interface #3 before unbinding 
<4> TRex interface #4 before unbinding 

Now choose the port you want to use and follow the next section by creating a configuration file.

==== Create minimum configuration file 

Create a configuration file in `/etc/trex_cfg.yaml`.

You could copy a basic configuration file from cfg folder by running this command.

[source,bash]
----
$cp  cfg/vm1.yaml /etc/trex_cfg.yaml
----

Now edit the configuration file with the right values from the previous section

[source,bash]
----
<none>
- port_limit    : 4    #<1>     
  version       : 2    #<2>
  interfaces    : ["03:00.0","03:00.1","13:00.1","13:00.0"]  #<3> 
----
<1> the number of ports 
<2> must add version 2 to the configuration file
<3> The list of interface from  `#>sudo ./dpdk_setup_ports.py -s`, in this example it was taken 

When working with VM, you must set the destination mac of one port as the source or the other for loopback the port in the vSwitch 
and you should take the right value from the hypervisor (in case of a physical NIC you can set the mac-address with virtual you can't and you should take it from the hypervisor)
and example

[source,python]
----
        - port_limit      : 2                                                                           
          version       : 2                                                                             
          interfaces    : ["03:00.0","03:00.1"]                                                         
          port_info       :  # set eh mac addr 
                  - dest_mac        :   [0x1,0x0,0x0,0x1,0x0,0x00]  # port 0                            
                    src_mac         :   [0x2,0x0,0x0,0x2,0x0,0x00]                                     <1>
                  - dest_mac        :   [0x2,0x0,0x0,0x2,0x0,0x00]  # port 1                           <1>
                    src_mac         :   [0x1,0x0,0x0,0x1,0x0,0x00]
----
<1>  source mac is like destination mac (this should be set or taken from vmware). the mac was taken from hypervisor 

==== Running TRex

Run this for 4x10Gb/sec TRex:
[source,bash]
----
$sudo ./t-rex-64 -f cap2/dns.yaml -c 4 -m 1 -d 100  -l 1000 
----

NOTE: For 10Gb/sec TRex with 2,6, or 8 ports, add --limit-ports [number of ports] *or* follow xref:trex_config[this] to configure the TRex. 
//TBD: recommend bold for the 2 commands.

If successful, the output will be similar to the following:

[source,python]
----
$ sudo ./t-rex-64 -f cap2/dns.yaml -d 100 -l 1000           
Starting  T-Rex 1.50 please wait  ... 
zmq publisher at: tcp://*:4500 
 number of ports founded : 4 
  port : 0 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex      <1>
  promiscuous  : 0 
  port : 1 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex
  promiscuous  : 0 
  port : 2 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex
  promiscuous  : 0 
  port : 3 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex
  promiscuous  : 0 


 -Per port stats table 
      ports |               0 |               1 |               2 |               3 
 -------------------------------------------------------------------------------------
   opackets |            1003 |            1003 |            1002 |            1002 
     obytes |           66213 |           66229 |           66132 |           66132 
   ipackets |            1003 |            1003 |            1002 |            1002 
     ibytes |           66225 |           66209 |           66132 |           66132 
    ierrors |               0 |               0 |               0 |               0 
    oerrors |               0 |               0 |               0 |               0 
      Tx Bw |     217.09 Kbps |     217.14 Kbps |     216.83 Kbps |     216.83 Kbps 

 -Global stats enabled 
 Cpu Utilization : 0.0  % <12>  29.7 Gb/core <13>
 Platform_factor : 1.0  
 Total-Tx        :     867.89 Kbps                                             <2>
 Total-Rx        :     867.86 Kbps                                             <3>
 Total-PPS       :       1.64 Kpps  
 Total-CPS       :       0.50  cps  

 Expected-PPS    :       2.00  pps   <9>
 Expected-CPS    :       1.00  cps   <10>
 Expected-BPS    :       1.36 Kbps   <11>

 Active-flows    :        0 <6> Clients :      510   Socket-util : 0.0000 %    
 Open-flows      :        1 <7> Servers :      254   Socket :        1 Socket/Clients :  0.0 
 drop-rate       :       0.00  bps   <8>
 current time    : 5.3 sec  
 test duration   : 94.7 sec  

 -Latency stats enabled 
 Cpu Utilization : 0.2 %  <14>
 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
 --------------------------------------------------------------------------------------------------
 0 |     1002,    1002,         0,   0,         51  ,      69,       0      |   0  69  67    <4>
 1 |     1002,    1002,         0,   0,         53  ,     196,       0      |   0  196  53   <5>
 2 |     1002,    1002,         0,   0,         54  ,      71,       0      |   0  71  69 
 3 |     1002,    1002,         0,   0,         53  ,     193,       0      |   0  193  52 
----
<1> Link must be up for TRex to work.
<2> Total Rx must be the same as Tx
<3> Total Rx must be the same as Tx 
<4> Tx_ok == Rx_ok
<5> Tx_ok == Rx_ok
<6> Number of TRex active "flows". Could be diffrent than the Router flows due to aging issues. Usualy TRex number of active flows is much lower that router.
<7> Number of TRex flows from startup.
<8> Drop rate.
<9> Expected Packet Per Second (without the latency packets).
<10> Expected Connection Per Second (without the latency packets).
<11> Expected Bit Per Second (without the latency packets).
<12> Average CPU utilization of transmitters threads. For best results it should be lower than 80%.
<13> Gb/sec generated per core of DP. Higer is better.
<14> Rx and latency thread CPU utilization.

WARNING: if you don't see rx packets, revisit your mac-address configuration.

==== Running TRex for the first time with router

You can follow this presentation link:trex_config_guide.html[first time TRex configuration] 
//TBD: Note that the link does not work correctly in PDF rendition
or continue reading. 
TRex set source-mac of all port to `00:00:00:01:00:00` and expected to get to this MAC-address `00:00:00:01:00:00` without a config file.
so you just need to configure router with the right mac-address.

NOTE: Virtual routers on ESXi (for example, Cisco CSR1000v) must have a distinct MAC address for each port. Specify the address in the configuration file. see more xref:trex_config[here]. Another example is where the TRex is connected to a switch. In that case each of TRex port should have a distinc MAC address.

include::trex_book_basic.asciidoc[]

== Advanced features


=== VLAN Trunk support anchor:trex_valn[]

The VLAN Trunk TRex feature attempts to solve the router port bandwidth limitation  when the traffic profile is asymmetric. Example: SFR profile is asymmetric and was the first usecase.
This feature converts asymmetric traffic to symmetric, from the port perspective, using router sub-interfaces.
This feature requires TRex to send the traffic on two VLANs. The following describes how this works.

.YAML format 
[source,python]
----
  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
----


.Example
[source,python]
----
- duration : 0.1
  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }   <1>
----
<1> enable VLAN feature , valn0==100 , valn1==200
        
*Problem definition:*::

Assuming a TRex with two ports and an SFR traffic profile.

.Without VLAN/sub interfaces 
[source,python]
----
0 ( client) -> [  ] - 1 ( server)
----
Without VLAN support it is not symmetric. From port 0 (client side),  it sends 10%, from and port 1 (server) sends 90%. Port 1 become the bottlneck (10Gb/s limit) before port 0 

.With VLAN/sub interfaces 
[source,python]
----
port 0 ( client VLAN0) <->  |  | <-> port 1 ( server-VLAN0)
port 0 ( server VLAN1) <->  |  | <-> port 1 ( client-VLAN1)
----

In this case both ports will have the same amount of traffic.

*Router configuation:*::
[source,python]
----
        !
        interface TenGigabitEthernet1/0/0      <1>
         mac-address 0000.0001.0000   
         mtu 4000
         no ip address
         load-interval 30
        !
        i
        interface TenGigabitEthernet1/0/0.100
         encapsulation dot1Q 100               <2> 
         ip address 11.77.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_100_p1_to_p2 <3> 
        !
        interface TenGigabitEthernet1/0/0.200
         encapsulation dot1Q 200               <4>
         ip address 11.88.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_200_p1_to_p2 <5> 
        !
        interface TenGigabitEthernet1/1/0
         mac-address 0000.0001.0000
         mtu 4000
         no ip address
         load-interval 30
        !
        interface TenGigabitEthernet1/1/0.100
         encapsulation dot1Q 100
         ip address 22.77.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_100_p2_to_p1
        !
        interface TenGigabitEthernet1/1/0.200
         encapsulation dot1Q 200
         ip address 22.88.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_200_p2_to_p1
        !
        
        arp 11.77.11.12 0000.0001.0000 ARPA      <6>
        arp 22.77.11.12 0000.0001.0000 ARPA
        
        route-map vlan_100_p1_to_p2 permit 10    <7>
         set ip next-hop 22.77.11.12
        !
        route-map vlan_100_p2_to_p1 permit 10
         set ip next-hop 11.77.11.12
        !
        
        route-map vlan_200_p1_to_p2 permit 10
         set ip next-hop 22.88.11.12
        !
        route-map vlan_200_p2_to_p1 permit 10
         set ip next-hop 11.88.11.12
        !
----
<1> Disable the IP on the main port it is important 
<2> Enable VLAN1
<3> PBR configuration
<4> Enable VLAN2  
<5> PBR configuration
<6> TRex MAC-address destination port 
<7> PBR configuration rules

=== Static source MAC-address setting  

With this feature, TRex replaces the source MAC address with the client IP address.
Note: This feature was requested by the Cisco ISG group. 


*YAML:*::
[source,python]
----
 mac_override_by_ip : true
----

.Example
[source,python]
----
- duration : 0.1
 ..
  mac_override_by_ip : true <1>
----
<1> In this case, the client side MAC address will be look like this:
SRC_MAC = IPV4(IP) + 00:00  

=== IPv6 support ( `--ipv6`);

Support for IPv6 includes:

1. Support for pcap files containing IPv6 packets
2. Ability to generate IPv6 traffic from pcap files containing IPv4 packets
The following switch enables this feature: `--ipv6`
Two new keywords (src_ipv6, dst_ipv6) have been added to the YAML
file to specify the most significant 96-bits of the IPv6 address - for example:

[source,python]
----
      src_ipv6 : [0xFE80,0x0232,0x1002,0x0051,0x0000,0x0000]
      dst_ipv6 : [0x2001,0x0DB8,0x0003,0x0004,0x0000,0x0000]
----
      
The IPv6 address is formed by placing what would typically be the IPv4
address into the least significant 32-bits and copying the value provided
in the src_ipv6/dst_ipv6 keywords into the most signficant 96-bits.
If src_ipv6 and dst_ipv6 are not specified in the YAML file, the default
is to form IPv4-compatible addresses (where the most signifcant 96-bits
are zero).

There is a support for all plugins (control flows that needed to be change).
  
*An example:*::
[source,bash]
----
$sudo ./t-rex-64 -f cap2l/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --ipv6
----

*Limitations:*::

* TRex cannot generate both IPv4 and IPv6 traffic. The --ipv6 switch must be specified even when using a pcap file containing only IPv6 packets


*Router configuration:*::

[source,python]
----
interface TenGigabitEthernet1/0/0
 mac-address 0000.0001.0000
 mtu 4000
 ip address 11.11.11.11 255.255.255.0
 ip policy route-map p1_to_p2
 load-interval 30
 ipv6 enable   ==> IPv6
 ipv6 address 2001:DB8:1111:2222::1/64                  <1>
 ipv6 policy route-map ipv6_p1_to_p2                    <2>
!


ipv6 unicast-routing                                    <3>

ipv6 neighbor 3001::2 TenGigabitEthernet0/1/0 0000.0002.0002   <4>
ipv6 neighbor 2001::2 TenGigabitEthernet0/0/0 0000.0003.0002

route-map ipv6_p1_to_p2 permit 10                              <5>
 set ipv6 next-hop 2001::2
!
route-map ipv6_p2_to_p1 permit 10
 set ipv6 next-hop 3001::2
!


asr1k(config)#ipv6 route 4000::/64 2001::2                 
asr1k(config)#ipv6 route 5000::/64 3001::2 
----
<1> enable ipv6 
<2> add pbr 
<3> enable ipv6 routing 
<4> mac-addr setting should be like T-Rex
<5> PBR configuraion


=== Source MAC-address mapping using a file 

Extending the source MAC-address  replacment capability. 
It is possible to have a mapping betwean IPv4->MAC using the new `--mac` CLI switch  
file format is YAML.

*An example:*::
[source,bash]
----
$sudo ./t-rex-64 -f cap2/sfr_delay_10_1g.yaml -c 4  -l 100 -d 100000 -m 30  --mac cap2/test_example.yaml
----

*MAC file structure:*::

[source,python]
----
- items :
    - ip : "16.0.0.1"
      mac : [0x16,0x1,0x4,0x5,0x6,0x7]
    - ip : "16.0.0.2"
      mac : [0x16,0x2,0x0,0x1,0x0,0x0]
----

*Limitations:*::

. It is assumed that most of the clients has MAC-addrees. at least 90% of the IP should have a MAC-addrees mapping.

=== Destination mac address spreadings anchor:mac_spread[] 

Using this option, one can send traffic to a few destination devices. In normal mode all the packets are sent to the port destination mac-address.
to enable this option add this CLI `--mac-spread` to the command line

example:

[source,bash]
----
$sudo ./t-rex-64 -f cap2/http_simple.yaml -d 1000 -m 1000 -c 4 -l 100 --mac-spread 2
----
in this case TRex will send to port destination mac and port destination mac +1 
using a switch you could connect TRex to a few DUT. 
All the DUTs should return the traffic only to right port source address

[source,bash]
----
        switch A                   switch A
         |                            |
         |   D0+0   -> DUT0  <- D1+0  |
TRex(0) -|                            |-TRex(1)
         |                            |
         |   D0+1  ->  DUT1  <- D1+1  |
                                      |

----


=== NAT support 

TRex can learn dynamic NAT/PAT translation. To enable this feature add `--learn` to the command line. 
//TBD: maybe... add the '--learn' option on the command line.
This is done by adding an IPv4 option header with TRex info (8 bytes long 0x10 id) to the first packet of the flow.

*Example:*::

*simple HTTP traffic*

[source,bash]
----
$sudo ./t-rex-64 -f cap2/http_simple.yaml -c 4  -l 1000 -d 100000 -m 30  --learn 
----

*SFR traffic without bundeling/ALG support*

[source,bash]
----
$sudo ./t-rex-64 -f avl/sfr_delay_10_1g_no_bundeling.yaml -c 4  -l 1000 -d 100000 -m 10  --learn 
----

*New terminal counters:*::

[source,python]
----
-Global stats enabled 
 Cpu Utilization : 0.6  %  33.4 Gb/core 
 Platform_factor : 1.0  
 Total-Tx        :     773.76 Mbps   Nat_time_out    :        0 <1>
 Total-Rx        :     770.47 Mbps   Nat_no_fid      :        0 <2>
 Total-PPS       :     106.73 Kpps   Total_nat_active:        9 <3>
 Total-CPS       :       2.78 Kcps   Total_nat_open  :   232129 <4>
----
<1> The number of translations with timeout should be zero. Usually this occurs when the router drops the flow due to NAT. 
<2> Translation not found. This can occur when there is large latency in the router input/output queue.
<3> Active number of TRex traslation flows, should be low in the case of low RTT.
<4> A total of TRex translation. May be different from the total number of flows in case template is uni-directional (no need a translation).


*Configuration for Cisco ASR1000 Series:*::

The feature was tested with the following configuration and  sfr_delay_10_1g_no_bundeling. yaml traffic profile.
Clients address range is 16.0.0.1-16.0.0.255 

[source,python]
----
interface TenGigabitEthernet1/0/0            <1>
 mac-address 0000.0001.0000
 mtu 4000
 ip address 11.11.11.11 255.255.255.0
 ip policy route-map p1_to_p2
 ip nat inside                               <2>
 load-interval 30
!

interface TenGigabitEthernet1/1/0
 mac-address 0000.0001.0000
 mtu 4000
 ip address 11.11.11.11 255.255.255.0
 ip policy route-map p1_to_p2
 ip nat outside                              <3>
 load-interval 30

ip  nat pool my 200.0.0.0 200.0.0.255 netmask 255.255.255.0  <4>

ip nat inside source list 7 pool my overload 
access-list 7 permit 16.0.0.0 0.0.0.255                      <5>

ip nat inside source list 8 pool my overload                 <6>
access-list 8 permit 17.0.0.0 0.0.0.255                      
----
<1> Should be connected to TRex Client port (router inside port)
<2> NAT inside 
<3> NAT outside
<4> Pool of outside address with overload
<5> Should match TRex YAML client range
<6> In case of dual port TRex.


*Limitations:*::

. The IPv6-IPv6 NAT feature does not exist on routers, so this feature can work on IPv4 only.
. Does not support NAT64. 
. Bundeling/plugin support is not fully supported. This means that sfr_delay_10.yaml can't work.Use sfr_delay_10_no_bundeling.yaml instead. 

[NOTE]
=====================================================================
* `--learn-verify` is a debug TRex mechanism for testing the TRex learn mechanism.
* If the router is configured without NAT, it will verify that the inside_ip==outside_ip and inside_port==outside_port.
=====================================================================

=== Flow order/latency verification ( `--rx-check` )

In normal mode (without this feature enabled), received traffic is not checked by software. It only counted by hardware (Intel NIC) for drop packets verification at test end of the test. The only exception is the Latency/Jitter packets.
This is one of the reasons that with TRex, you *cannot* check features that terminate traffic (for example TCP Proxy)
To enable this feature you should add to the command line options `--rx-check [sample]` where sample is the sample rate. 
1/sample flows will be loaded to the software for verification. For 40Gb/Sec traffic you can use a sample of 1/128. Watch for Rx CPU% utilization.

INFO : This feature changes the TTL of the sample flows to 255 and expects 254 or 255 (one routing hop). If you have more than one hop in your setup, use `--hops` to change it to higher value. more than one hop could happned when there are number of routers betwean TRex client side to TRex server side.

With this feature enabled:

* You can verify that packets get out of DUT in order (from each flow perspective)
* There are no dropped packets. There is no need to wait for the end of the test. Without this feature enabled you must wait for the end of the test to be aware of dropped packets  because there is always a difference between TX and Rx due RTT. To be sure there is a need to stop the traffic and wait for the Rx traffic and this happens only at the end of the test.


.Full example 
[source,bash]
----
$sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --rx-check 128
----

[source,python]
----
Cpu Utilization : 0.1 %                                                                       <1>
 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter<1> ,  max window 
   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
 --------------------------------------------------------------------------------
 0 |     1002,    1002,      2501,   0,         61  ,      70,       3      |  60
 1 |     1002,    1002,      2012,   0,         56  ,      63,       2      |  50
 2 |     1002,    1002,      2322,   0,         66  ,      74,       5      |  68
 3 |     1002,    1002,      1727,   0,         58  ,      68,       2      |  52

 Rx Check stats enabled                                                                       <2>
 -------------------------------------------------------------------------------------------
 rx check:  avg/max/jitter latency,       94  ,     744,       49<1>      |  252  287  309    <3>
 
 active flows:       10, fif:      308,  drop:        0, errors:        0                     <4>
 -------------------------------------------------------------------------------------------
----
<1> CPU% of the Rx thread. If it is too high *increase* the sample rate.
<2> Rx Check section. For more detailed info, press 'r' during the test or at the end of the test.
<3> Average latency, max latency, jitter on the template flows in microseconds. This is usually *higher* than the latency check packet because the feature works more on this packet.
<4> Drop counters and errors counter should be zero. If not, press 'r' to see the full report or view the report at the end of the test.

.Full report by pressing 'r'
[source,python]
----
 m_total_rx                              : 2 
 m_lookup                                : 2 
 m_found                                 : 1 
 m_fif                                   : 1 
 m_add                                   : 1 
 m_remove                                : 1 
 m_active                                : 0 
                                                        <1>
 0  0  0  0  1041  0  0  0  0  0  0  0  0  min_delta  : 10 usec 
 cnt        : 2 
 high_cnt   : 2 
 max_d_time : 1041 usec
 sliding_average    : 1 usec
 precent    : 100.0 %
 histogram 
 -----------
 h[1000]  :  2 
 tempate_id_ 0 , errors:       0,  jitter: 61           <2>
 tempate_id_ 1 , errors:       0,  jitter: 0 
 tempate_id_ 2 , errors:       0,  jitter: 0 
 tempate_id_ 3 , errors:       0,  jitter: 0 
 tempate_id_ 4 , errors:       0,  jitter: 0 
 tempate_id_ 5 , errors:       0,  jitter: 0 
 tempate_id_ 6 , errors:       0,  jitter: 0 
 tempate_id_ 7 , errors:       0,  jitter: 0 
 tempate_id_ 8 , errors:       0,  jitter: 0 
 tempate_id_ 9 , errors:       0,  jitter: 0 
 tempate_id_10 , errors:       0,  jitter: 0 
 tempate_id_11 , errors:       0,  jitter: 0 
 tempate_id_12 , errors:       0,  jitter: 0 
 tempate_id_13 , errors:       0,  jitter: 0 
 tempate_id_14 , errors:       0,  jitter: 0 
 tempate_id_15 , errors:       0,  jitter: 0 
 ager :
 m_st_alloc                                 : 1 
 m_st_free                                  : 0 
 m_st_start                                 : 2 
 m_st_stop                                  : 1 
 m_st_handle                                : 0 
----
<1> Any errors shown here
<2> Error per template info

*Limitation:*::

** This feature must be enabled with a latency check (-l).
** To receive the packets TRex does the following:
*** Changes the TTL to 0xff and expects 0xFF (loopback) or oxFE (route). ( use --hop to tune this number)
*** Adds 24 bytes of metadata as ipv4/ipv6 option header   

== Reference

=== Traffic YAML

==== Global Traffic YAML section 

[source,python]
----
- duration : 10.0                          <1>
  generator :                              <2>
          distribution : "seq"           
          clients_start : "16.0.0.1"     
          clients_end   : "16.0.0.255"   
          servers_start : "48.0.0.1"     
          servers_end   : "48.0.0.255"   
          clients_per_gb : 201
          min_clients    : 101
          dual_port_mask : "1.0.0.0" 
          tcp_aging      : 1
          udp_aging      : 1
  mac        : [0x00,0x00,0x00,0x01,0x00,0x00] <3>
  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 } <7>
  mac_override_by_ip : true  <8>
  cap_ipg    : true                            <4>
  cap_ipg_min    : 30                          <5>
  cap_override_ipg    : 200                    <6>
----
<1> Duration of the test (seconds). Can override using the `-d` option.
<2> See the generator section.
<3> Default source/destination MAC address. The configuration file can override the defaults.
<4> TRUE indicates that the IPG is taken from pcap file.
<5> The following two options can set the min ipg in microseconds: ( if (pkt_ipg<cap_ipg_min) { pkt_ipg=cap_override_ipg) }
<6> Value to override (microseconds). 
<7> Enable valn feature. See xref:trex_valn[here] for info.
<8> Enable MAC address replacement by Client IP.

==== Per template section 

[source,python]
----
     - name: cap2/dns.pcap <1>
       cps : 10.0          <2>
       ipg : 10000         <3>
       rtt : 10000         <4>
       w   : 1             <5>
       server_addr : "48.0.0.7"    <6>
       one_app_server : true       <7>
       
----
<1> The name of the template pcap file. It can be relative to the t-rex-64 image or absolute path. The pcap file can include one flow. (Exception: in case of plug-ins).
<2> Connection per second for m==1
<3> If the global section of the YAML file does not include `cap_ipg    : true`, this line sets the inter-packet gap in microseconds. 
<4> Should be set to the same value as ipg (microseconds). 
<5> Default value: w=1. This indicates to the IP generator how to generate the flows. If w=2, two flows from the same template will be generated in a burst (more for HTTP that has burst of flows).
<6> If `one_app_server` is set to true, then all templates will use the same server.
<7> If the same server address is required, set this value to true.



=== Configuration YAML anchor:trex_config[]

The configuration file, in YAML format, configures TRex behavior, including:

 
- MAC address for each port (source and destination)
- Masking interfaces (usually for 1Gb/Sec TRex) to ensure that TRex does not take the management ports as traffic ports.
- Changing the zmq/telnet TCP port.

==== Basic Configuration 

Copy/install the configuration file to `/etc/trex_cfg.yaml`. 
TRex loads it automatically at startup. You still can override options with the command line option switch `--cfg [file]` in the CLI
Configuration file examples can be found in the `$ROOT/cfg` folder


[source,python]
----
        - port_limit      : 2                                                                           <1>
          version       : 2                                                                             <2>
          interfaces    : ["03:00.0","03:00.1"]                                                         <3>          
          enable_zmq_pub  : true                                                                        <4>
          zmq_pub_port    : 4500                                                                        <5>
          prefix          : setup1                                                                      <6>
          limit_memory    : 1024                                                                        <7>
          c               : 4                                                                           <8>
          port_bandwidth_gb : 10                                                                        <9>
          port_info       :  # set eh mac addr 
                  - dest_mac        :   [0x1,0x0,0x0,0x1,0x0,0x00]  # port 0                            <10>
                    src_mac         :   [0x2,0x0,0x0,0x2,0x0,0x00]
                  - dest_mac        :   [0x3,0x0,0x0,0x3,0x0,0x00]  # port 1
                    src_mac         :   [0x4,0x0,0x0,0x4,0x0,0x00]
                  - dest_mac        :   [0x5,0x0,0x0,0x5,0x0,0x00]  # port 2
                    src_mac         :   [0x6,0x0,0x0,0x6,0x0,0x00]
                  - dest_mac        :   [0x7,0x0,0x0,0x7,0x0,0x01]  # port 3
                    src_mac         :   [0x0,0x0,0x0,0x8,0x0,0x02]
                  - dest_mac        :   [0x0,0x0,0x0,0x9,0x0,0x03]  # port 4
----
<1>  The number of ports, should be equal to the number of interfaces in 3)
<2>  Must be set to 2
<3>  Interface that should be used. used `sudo ./dpdk_setup_ports.py --show`
<4>  Enable the ZMQ publisher for stats data.
<5>  ZMQ port number.

<6> The name of the setup should be distinct ( DPDK --file-prefix )
<7> DPDK -m limit the packet memory 
<8> Number of threads per dual interface ( like -c CLI option )
<9> The bandwidth of each interface in Gb/sec. In this example we have 10Gb/sec interfaces. for VM put 1. it used to tune the amount of memory allocated by TRex.
<10>  MAC address per port - source and destination.


To find out what the interfaces ids, perform the following:

[source,bash]
----
 $>sudo ./dpdk_setup_ports.py --show

 Network devices using DPDK-compatible driver
 ============================================

 Network devices using kernel driver
 ===================================
 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active*
 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<1>
 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<2>
 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<3>
 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<4>

 Other network devices
 =====================
 <none>
----
<1> TRex interface #1 before unbinding 
<2> TRex interface #2 before unbinding 
<3> TRex interface #3 before unbinding 
<4> TRex interface #4 before unbinding 


minimum configuration file is:

[source,bash]
----
<none>
- port_limit    : 4         
  version       : 2   #<1>
  interfaces    : ["03:00.0","03:00.1","13:00.1","13:00.0"]  #<2> 
----
<1> must add version 2 to the configuration file
<2> The list of interfaces from `sudo ./dpdk_setup_ports.py --show`


==== Memory section configuration  

The memory section is optional. It is used when there is a need to tune the amount of memory used by packet manager 

[source,python]
----
        - port_limit      : 2                                                                           
          version       : 2                                                                             
          interfaces    : ["03:00.0","03:00.1"]                                                         
          memory    :                                           <1>
             mbuf_64     : 16380                                <2>
             mbuf_128    : 8190
             mbuf_256    : 8190
             mbuf_512    : 8190
             mbuf_1024   : 8190
             mbuf_2048   : 4096
             traffic_mbuf_64     : 16380                        <3>
             traffic_mbuf_128    : 8190
             traffic_mbuf_256    : 8190
             traffic_mbuf_512    : 8190
             traffic_mbuf_1024   : 8190
             traffic_mbuf_2048   : 4096
             dp_flows    : 1048576                              <4>
             global_flows : 10240                               <5>
----
<1> Memory section 
<2> Per dual interfaces number of buffers - buffer for real time traffic generation 
<3> Traffic buffer - when you have many template only this section should be enlarge  
<4> number of TRex flows needed 
<5> reserved 


==== Platform section configuration  

The platform section is optional. It is used to tune the performance and allocate the cores to the right NUMA  
a configuration file now has the folowing struct to support multi instance 

[source,python]
----
- version       : 2
  interfaces    : ["03:00.0","03:00.1"]                         
  port_limit    : 2 
  enable_zmq_pub  : true  # enable publisher for stats data 
  zmq_pub_port    : 4507                                        
  prefix          : setup1                                      <1>
  limit_memory    : 1024                                        <2>
  c               : 4                                           <3>
  port_bandwidth_gb : 10                                        <4>
  platform :                                                    <5>
        master_thread_id  : 0                                   <6>        
        latency_thread_id : 5                                   <7>
        dual_if   :                                             
             - socket   : 0                                     <8>
               threads  : [1,2,3,4]                             <9>
----
<1> The name of the setup should be distinct ( DPDK --file-prefix )
<2> DPDK -m 
<3> Number of threads per dual interface ( like -c CLI option )
<4> The bandwidth of each interface in Gb/sec. In this example we have 10Gb/sec interfaces. for VM put 1. it used to tune the amount of memory allocated by TRex.
<5> the platform section
<6> The thread_id for control 
<7> The thread_id for latency if used
<8> Socket of the dual interfaces, in this example of 03:00.0 and 03:00.1, memory should be local to the interface
<9> Thread to be used, should be local to the NIC 



=== Command line options anchor:cml-line[]

*-f=TRAFIC_YAML_FILE*:: 
       Traffic YAML configuration file.

*-c=CORES*::
  Number of cores. Use 4 for TRex 40Gb/sec. Monitor the CPU% of TRex - it should be ~50%.

*-l=HZ*::
    Run the latency daemon in this Hz rate. Example: -l 1000 runs 1000 pkt/sec from each interface. A value of zero (0) disables the latency check.

*-d=DURATION*::
        Duration of the test (sec), Default: 0 

*-m=MUL*::
   Factor for bandwidth (multiply the CPS of each template by this value).

*--ipv6*::
       Convert template to IPv6 mode.

*--learn*::                   
    Learn the dynamic NAT translation and ALG.
    
*--learn-verify*::          
    Learn the translation. This feature is intended for verification of the mechanism in cases where there is no NAT.
  
*-p*::                
  Flow-flip. Sends all flow packets from the same interface. This can solve the flow order. Does not work with any router configuration.

*-e*::                
  same as `-p` but comply to the direction rules and replace source/destination IPs. it might not be good for NBAR as it is expected clients ip to be sent from same direction.

//TBD: The last 2 sentences (flow order, router configuration) are unclear.

  
*--lm=MASK*::                        
  Latency mask. Use this to verify port connectivity. Possible values: 0x1 (only port 0 will send traffic), 0x2 (only port 1 will send traffic). 

*--lo*::                         
   Latency test.
  
*--limit-ports=PORTS*::
   Limit number of ports. Configure this in the --cfg file. Possible values (number of ports): 2, 4, 6, 8. (Default: 4)
  
*--nc*::       
    If set, will terminate exacly at the end of the duration. This provides a faster, more accurate TRex termination. In default it wait for all the flow to terminate gracefully. In case of a very long flow the termination might be prolong. 
    
*-pm=MULTIFLIER*::
   Platform factor. If the setup includes a splitter, you can multiply the total results by this factor. Example: --pm 2.0 will multiply all bps results by this factor.
  
*-pubd*::
  Disable ZMQ monitor's  publishers.

*-1g*::                       
   Deprecated. Configure TRex to 1G. Configure this in the --cfg file. 
  
*-k=KSEC*::                  
   Run a latency test before starting the test. TRex will wait for x sec before and after sending latency packets at startup.
  
*--cfg=platform_yaml*::     
        Load and configure platform using this file. See example file: cfg/cfg_examplexx.yaml 
        This file is used to configure/mask interfaces, cores, affinity, and MAC addresses.  
        You can use the example file by copying it to: /etc/trex_cfg.yaml   
  
  
*-v=VERBOSE*::
   Verbose mode (works only on the debug image! )  
      1    Show only stats.
      2    Run preview. Does not write to file.
      3    Run preview and write to stats file.
  Note: When using verbose mode, it is not necessary to add an output file. 
  Caution: Operating in verbose mode can generate very large files (terabytes). Use with caution, only on a local drive.
 
  
*--rx-check=SAMPLE_RATE*::
        Enable Rx check module. Using this each thread samples flows (1/sample) and checks order, latency, and additional statistics.
        Note: This feature operates as an additional thread.
  
*--hops=HOPES*::
   Number of hops in the setup (default is one hop). Relevant only if the Rx check is enabled.
   
*--iom=MODE*::
        I/O mode for interactive mode. Possible values: 0 (silent), 1 (normal), 2 (short)
  
*--no-flow-control*::
  Prevents TRex from changing flow control. In default TRex operation, flow control is disabled at startup.

*--mac-spread*::            
  Spread the destination mac by this this factor. e.g 2 will generate the traffic to 2 devices DEST-MAC ,DEST-MAC+1. The maximum is up to 128 devices.


ifndef::backend-docbook[]


endif::backend-docbook[]

== Appendix

=== Simulator 
 
The TRex simulator is a linux application that can process on any Linux CEL (it can run on TRex itself).
you can create create output pcap file from input of traffic YAML.

====  Simulator 


[source,bash]
----

$./bp-sim-64-debug -f avl/sfr_delay_10_1g.yaml -v 1

 -- loading cap file avl/delay_10_http_get_0.pcap 
 -- loading cap file avl/delay_10_http_post_0.pcap 
 -- loading cap file avl/delay_10_https_0.pcap 
 -- loading cap file avl/delay_10_http_browsing_0.pcap 
 -- loading cap file avl/delay_10_exchange_0.pcap 
 -- loading cap file avl/delay_10_mail_pop_0.pcap 
 -- loading cap file avl/delay_10_mail_pop_1.pcap 
 -- loading cap file avl/delay_10_mail_pop_2.pcap 
 -- loading cap file avl/delay_10_oracle_0.pcap 
 -- loading cap file avl/delay_10_rtp_160k_full.pcap 
 -- loading cap file avl/delay_10_rtp_250k_full.pcap 
 -- loading cap file avl/delay_10_smtp_0.pcap 
 -- loading cap file avl/delay_10_smtp_1.pcap 
 -- loading cap file avl/delay_10_smtp_2.pcap 
 -- loading cap file avl/delay_10_video_call_0.pcap 
 -- loading cap file avl/delay_10_sip_video_call_full.pcap 
 -- loading cap file avl/delay_10_citrix_0.pcap 
 -- loading cap file avl/delay_10_dns_0.pcap 
 id,name                                    , tps, cps,f-pkts,f-bytes, duration,   Mb/sec,   MB/sec,   c-flows,  PPS,total-Mbytes-duration,errors,flows    #<2>
 00, avl/delay_10_http_get_0.pcap             ,404.52,404.52,    44 ,   37830 ,   0.17 , 122.42 ,   15.30 ,         67 , 17799 ,       2 , 0 , 1 
 01, avl/delay_10_http_post_0.pcap            ,404.52,404.52,    54 ,   48468 ,   0.21 , 156.85 ,   19.61 ,         85 , 21844 ,       2 , 0 , 1 
 02, avl/delay_10_https_0.pcap                ,130.87,130.87,    96 ,   91619 ,   0.22 ,  95.92 ,   11.99 ,         29 , 12564 ,       1 , 0 , 1 
 03, avl/delay_10_http_browsing_0.pcap        ,709.89,709.89,    37 ,   34425 ,   0.13 , 195.50 ,   24.44 ,         94 , 26266 ,       2 , 0 , 1 
 04, avl/delay_10_exchange_0.pcap             ,253.81,253.81,    43 ,    9848 ,   1.57 ,  20.00 ,    2.50 ,        400 , 10914 ,       0 , 0 , 1 
 05, avl/delay_10_mail_pop_0.pcap             ,4.76,4.76,    20 ,    5603 ,   0.17 ,   0.21 ,    0.03 ,          1 ,    95 ,       0 , 0 , 1 
 06, avl/delay_10_mail_pop_1.pcap             ,4.76,4.76,   114 ,  101517 ,   0.25 ,   3.86 ,    0.48 ,          1 ,   543 ,       0 , 0 , 1 
 07, avl/delay_10_mail_pop_2.pcap             ,4.76,4.76,    30 ,   15630 ,   0.19 ,   0.60 ,    0.07 ,          1 ,   143 ,       0 , 0 , 1 
 08, avl/delay_10_oracle_0.pcap               ,79.32,79.32,   302 ,   56131 ,   6.86 ,  35.62 ,    4.45 ,        544 , 23954 ,       0 , 0 , 1 
 09, avl/delay_10_rtp_160k_full.pcap          ,2.78,8.33,  1354 , 1232757 ,  61.24 ,  27.38 ,    3.42 ,        170 ,  3759 ,       0 , 0 , 3 
 10, avl/delay_10_rtp_250k_full.pcap          ,1.98,5.95,  2069 , 1922000 ,  61.38 ,  30.48 ,    3.81 ,        122 ,  4101 ,       0 , 0 , 3 
 11, avl/delay_10_smtp_0.pcap                 ,7.34,7.34,    22 ,    5618 ,   0.19 ,   0.33 ,    0.04 ,          1 ,   161 ,       0 , 0 , 1 
 12, avl/delay_10_smtp_1.pcap                 ,7.34,7.34,    35 ,   18344 ,   0.21 ,   1.08 ,    0.13 ,          2 ,   257 ,       0 , 0 , 1 
 13, avl/delay_10_smtp_2.pcap                 ,7.34,7.34,   110 ,   96544 ,   0.27 ,   5.67 ,    0.71 ,          2 ,   807 ,       0 , 0 , 1 
 14, avl/delay_10_video_call_0.pcap           ,11.90,11.90,  2325 , 2532577 ,  36.56 , 241.05 ,   30.13 ,        435 , 27662 ,       3 , 0 , 1 
 15, avl/delay_10_sip_video_call_full.pcap    ,29.35,58.69,  1651 ,  120315 ,  24.56 ,  28.25 ,    3.53 ,        721 , 48452 ,       0 , 0 , 2 
 16, avl/delay_10_citrix_0.pcap               ,43.62,43.62,   272 ,   84553 ,   6.23 ,  29.51 ,    3.69 ,        272 , 11866 ,       0 , 0 , 1 
 17, avl/delay_10_dns_0.pcap                  ,1975.02,1975.02,     2 ,     162 ,   0.01 ,   2.56 ,    0.32 ,         22 ,  3950 ,       0 , 0 , 1 

 00, sum                                      ,4083.86,93928.84,  8580 , 6413941 ,   0.00 , 997.28 ,  124.66 ,       2966 , 215136 ,      12 , 0 , 23 
 Memory usage 
 size_64        : 1687 
 size_128       : 222 
 size_256       : 798 
 size_512       : 1028 
 size_1024      : 86 
 size_2048      : 4086 
 Total    :       8.89 Mbytes  159% util #<1>

----
<1> the memory usage of the templates 
<2> CSV for all the templates