path: root/trex_book.asciidoc

TRex 
====
:author: hhaim 
:email: <hhaim@cisco.com> 
:revnumber: 2.1
:quotes.++:
:numbered:
:web_server_url: http://trex-tgn.cisco.com/trex
:local_web_server_url: csi-wiki-01:8181/trex
:toclevels: 4

include::trex_ga.asciidoc[]


== Introduction

=== A word on traffic generators

Traditionally, routers have been tested using commercial traffic generators, while performance
typically has been measured using packets per second (PPS) metrics. As router functionality and
services have become more complex, stateful traffic generators have become necessary to
provide more realistic application traffic scenarios.

Advantages of realistic traffic generators:

* Accurate performance metrics
* Discovering bottlenecks in realistic traffic scenarios

==== Current Challenges:

* *Cost*: Commercial stateful traffic generators are expensive
* *Scale*: Bandwidth does not scale up well with feature complexity
* *Standardization*: Lack of standardization of traffic patterns and methodologies
* *Flexibility*: Commercial tools do not allow agility when flexibility and changes are needed

==== Implications 

* High capital expenditure (capex) spent by different teams.
* Testing in low scale and extrapolation became a common practice. This is non-ideal and fails to indicate bottlenecks that appear in real-world scenarios.
* Teams use different benchmark methodologies, so results are not standardized.
* Delays in development and testing due to dependence on testing tool features.
* Resource and effort investment in developing different ad hoc tools and test methodologies.

=== Overview of TRex

TRex addresses these problems through an innovative and extendable software implementation and by leveraging standard and open software and x86/UCS hardware.

* Generates and analyzes L4-7 traffic. In one package, provides capabilities of commercial L7 tools.
* Stateful traffic generator based on pre-processing and smart replay of real traffic templates.
* Generates and *amplifies* both client and server side traffic.
* Customized functionality can be added.
* Scales to 200Gb/sec for one UCS (using Intel 40Gb/sec NICs)
* Low cost
* Self-contained package that can be easily installed and deployed
* Virtual interface support enables TRex to be used in a fully virtual environment without physical NICs. Example use cases:
** Amazon AWS
** Cisco LaaS
// Which LaaS is this? Location as a service? Linux?
** TRex on your laptop



.TRex Hardware 
[options="header",cols="1^,1^"]
|=================
|Cisco UCS Platform | Intel NIC 
| image:images/ucs200_2.png[title="generator"] | image:images/Intel520.png[title="generator"]  
|=================

=== Purpose of this guide

This guide explains the use of TRex internals and the use of TRex together with Cisco ASR1000 Series routers. The examples illustrate novel traffic generation techniques made possible by TRex. 

== Download and installation

=== Hardware recommendations

TRex operates in a Linux application environment, interacting with Linux kernel modules. 
TRex curretly works on x86 architecture and can operate well on Cisco UCS hardware. The following platforms have been tested and are recommended for operating TRex. 

[NOTE]
=====================================
 A high-end UCS platform is not required for operating TRex in its current version, but may be required for future versions.
=====================================

[NOTE]
=====================================
 Not all supported DPDK interfaces are supported by TRex
=====================================
 

.Preferred UCS hardware
[options="header",cols="1,3"]
|=================
| UCS Type | Comments 
| UCS C220 M3/M4  | *Preferred Low-End*. Supports up to 40Gb/sec with 540-D2. With newer Intel NIC (recommended), supports 80Gb/sec with 1RU. See table below describing components.
| UCS C200| Early UCS model.
| UCS C210 M2 | Supports up to 40Gb/sec PCIe3.0.
| UCS C240 M3/M4 | *Preferred, High-End* Supports up to 200Gb/sec. 6x XL710 NICS (PCIex8) or 2xFM10K (PCIex16). See table below describing components.
| UCS C260M2 | Supports up to 30Gb/sec (limited by V2 PCIe).
|=================

.Low-End UCS C220 M4 - Internal components
[options="header",cols="1,2",width="60%"]
|=================
| Components |  Details 
| CPU  | 2x E5-2620 @ 2.0 GHz. 
| CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
| Memory | 2x4 banks f.or each CPU. Total of 32GB in 8 banks.	 
| RAID | No RAID.
|=================

.High-End C240 M4 - Internal components 
[options="header",cols="1,2",width="60%"]
|=================
| Components |  Details 
| CPU  | 2x E5-2667 @ 3.20 GHz.
| PCIe | 1x Riser PCI expansion card option A PID UCSC-PCI-1A-240M4 enables 2 PCIex16.
| CPU Configuration | 2-Socket CPU configurations (also works with 1 CPU).
| Memory | 2x4 banks for each CPU. Total of 32GB in 8 banks. 
| RAID | No RAID.
|=================
 
.Supported NICs
[options="header",cols="1,1,2",width="50%"]
|=================
| Bandwidth | Chipset |  Example
| 1Gb/sec  | Intel I350 | Intel 4x1GE 350-T4 NIC
| 10Gb/sec | Intel 82599| Intel x520-D2 Cisco Order tool 2X Intel N2XX-AIPCI01, Intel X520 Dual Port 10Gb SFP+ Adapter
| 10Gb/sec | Intel X710   | link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+], *Preferred* support per stream stats in hardware link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[Silicom PE310G4i71L]
| 40Gb/sec | Intel XL710  | link:https://en.wikipedia.org/wiki/QSFP[QSFP+] (copper/optical)
| 100Gb/sec | Intel Intel FM10420  | QSFP28,  by Silicom link:http://www.silicom-usa.com/100_Gigabit_Dual_Port_Fiber_Ethernet_PCI_Express_PE3100G2DQiR_96[Silicom PE3100G2DQiR_96] (*in development*)
| VMXNET / +
VMXNET3 (see notes) | VMware paravirtualized  | Connect using VMware vSwitch
| E1000    | paravirtualized  | VMware/KVM/VirtualBox 
|=================

// in table above, is it correct to list "paravirtualized" as chipset? Also, what is QSFP28? It does not appear on the lined URL. Clarify: is Intel X710 the preferred NIC?

.X710 NIC base  SFP+ support 
[options="header",cols="1,1,1",width="70%"]
|=================
| link:https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver[SFP+]  | Intel Ethernet Converged X710-DAX |  Silicom link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[PE310G4i71L] (Open optic)
| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-SR] | Does not work     | [green]*works*
| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-LR] | Does not work     | [green]*works*
| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-H10GB-CU1M]| [green]*works* | [green]*works*
| link:http://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_c78-455693.html[Cisco SFP-10G-AOC1M] | [green]*works* | [green]*works*
|=================

[NOTE]
=====================================
 Intel X710 NIC (example: FH X710DA4FHBLK) operates *only* with Intel SFP+. For open optic, use the link:http://www.silicom-usa.com/PE310G4i71L_Quad_Port_Fiber_SFP+_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_49[Silicom PE310G4i71L] NIC.
=====================================

// clarify above table and note

.XL710 NIC base QSFP+ support 
[options="header",cols="1,1,1",width="90%"]
|=================
| link:https://en.wikipedia.org/wiki/QSFP[QSFP+]             |  Intel Ethernet Converged XL710-QDAX | Silicom link:http://www.silicom-usa.com/Dual_Port_Fiber_40_Gigabit_Ethernet_PCI_Express_Server_Adapter_PE340G2Qi71_83[PE340G2Qi71] Open optic 
| QSFP+ SR4 optics  |  APPROVED OPTICS [green]*works*,  Cisco QSFP-40G-SR4-S does *not* work   | Cisco QSFP-40G-SR4-S [green]*works* 
| QSFP+ LR-4 Optics |   APPROVED OPTICS [green]*works*, Cisco QSFP-40G-LR4-S does *not* work   | Cisco QSFP-40G-LR4-S [green]*works* 
| QSFP Active Optical Cables (AoC) | Cisco QSFP-H40G-AOC [green]*works*  | Cisco QSFP-H40G-AOC [green]*works* 
| QSFP+ Intel Ethernet Modular Optics |    N/A            |  N/A
| QSFP+ DA twin-ax cables | N/A  | N/A  
| Active QSFP+ Copper Cables | Cisco QSFP-4SFP10G-CU [green]*works*  | Cisco QSFP-4SFP10G-CU [green]*works*
|=================

[NOTE]
=====================================
 For Intel XL710 NICs, Cisco SR4/LR QSFP+ does not operate. Use Silicom with Open Optic.
=====================================

// clarify above table and note. let's discuss.

.FM10K QSFP28 support 
[options="header",cols="1,1",width="70%"]
|=================
| QSFP28             | Example 
| todo  |  todo
|=================

// do we want to show "todo"? maybe "pending"


[IMPORTANT]
=====================================
* Intel SFP+ 10Gb/sec is the only one supported by default on the standard Linux driver. TRex also supports Cisco 10Gb/sec SFP+. 
// above, replace "only one" with "only mode"?
* For operating high speed throughput (example: several Intel XL710 40Gb/sec), use different link:https://en.wikipedia.org/wiki/Non-uniform_memory_access[NUMA] nodes for different NICs. +
    To verify NUMA and NIC topology: `lstopo (yum install hwloc)` +
    To display CPU info, including NUMA node: `lscpu` +
    NUMA usage xref:numa-example[example]
* For Intel XL710 NICs, verify that the NVM is v5.04 . xref:xl710-firmware[Info].
**  `> sudo ./t-rex-64 -f cap2/dns.yaml -d 0 *-v 6* --nc | grep NVM` +
    `PMD:  FW 5.0 API 1.5 NVM 05.00.04  eetrack 800013fc` 
=====================================

// above, maybe rename the bullet points "NIC usage notes"? should we create a subsection for NICs? Maybe it would be under "2.1 Hardware recommendations" as a subsection.


.Sample order for recommended low-end Cisco UCSC-C220-M3S with 4x10Gb ports
[options="header",cols="1,1",width="70%"]
|=================
| Component | Quantity 
| UCSC-C220-M3S    |  1
| UCS-CPU-E5-2650  |  2
| UCS-MR-1X041RY-A |  8
| A03-D500GC3      |  1
| N2XX-AIPCI01     |  2
| UCSC-PSU-650W    |  1
| SFS-250V-10A-IS  |  1
| UCSC-CMA1        |  1
| UCSC-HS-C220M3   |  2
| N20-BBLKD        |  7
| UCSC-PSU-BLKP    |  1
| UCSC-RAIL1       |  1
|=================

// should table above say "low-end Cisco UCS C220 M3S" instead of "low-end USCS-C220-M3S"?

NOTE: Purchase the 10Gb/sec SFP+ separately. Cisco would be fine with TRex (but not for plain Linux driver).
// does note above mean "TRex operates with 10Gb/sec SFP+ components, but plain Linux does not provide drivers."? if so, how does purchasing separately solve this? where do they get drivers?

=== Installing OS 

==== Supported versions

Supported Linux versions:

* Fedora 20-23, 64-bit kernel (not 32-bit)
* Ubuntu 14.04.1 LTS, 64-bit kernel (not 32-bit)
* Ubuntu 16.xx LTS, 64-bit kernel (not 32-bit) 

NOTE: Additional OS version may be supported by compiling the necessary drivers.

To check whether a kernel is 64-bit, verify that the ouput of the following command is `x86_64`.

[source,bash]
----
$uname -m
x86_64 
----


==== Download Linux

ISO images for supported Linux releases can be downloaded from:

.Supported Linux ISO image links
[options="header",cols="1^,2^",width="50%"]
|======================================
| Distribution | SHA256 Checksum
| link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-DVD.iso[Fedora 20]
    | link:http://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Fedora/x86_64/iso/Fedora-20-x86_64-CHECKSUM[Fedora 20 CHECKSUM] 
| link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-DVD-x86_64-21.iso[Fedora 21]
    | link:http://fedora-mirror01.rbc.ru/pub/fedora/linux/releases/21/Server/x86_64/iso/Fedora-Server-21-x86_64-CHECKSUM[Fedora 21 CHECKSUM] 
| link:http://old-releases.ubuntu.com/releases/14.04.1/ubuntu-14.04-desktop-amd64.iso[Ubuntu 14.04.1]
    | http://old-releases.ubuntu.com/releases/14.04.1/SHA256SUMS[Ubuntu 14.04 CHECKSUM]
|======================================

For Fedora downloads...

* Select a mirror close to your location: +
https://admin.fedoraproject.org/mirrormanager/mirrors/Fedora +
Choose: "Fedora Linux http" -> releases -> <version number> -> Server -> x86_64 -> iso -> Fedora-Server-DVD-x86_64-<version number>.iso

* Verify the checksum of the downloaded file matches the linked checksum values with the `sha256sum` command. Example:

[source,bash]
----
$sha256sum Fedora-18-x86_64-DVD.iso
91c5f0aca391acf76a047e284144f90d66d3d5f5dcd26b01f368a43236832c03 #<1>
----
<1> Should be equal to the link:https://en.wikipedia.org/wiki/SHA-2[SHA-256] values described in the linked checksum files. 


==== Install Linux 

Ask your lab admin to install the Linux using CIMC, assign an IP, and set the DNS. Request the sudo or super user password to enable you to ping and SSH.

xref:fedora21_example[Example of installing Fedora 21 Server]

[NOTE]
=====================================
 * To use TRex, you should have sudo on the machine or the root password.
 * Upgrading the linux Kernel using `yum upgrade` requires building the TRex drivers. 
=====================================

==== Verify Intel NIC installation

Use `lspci` to verify the NIC installation. 

Example 4x 10Gb/sec TRex configuration (see output below):

* I350 management port

* 4x Intel Ethernet Converged Network Adapter model x520-D2 (82599 chipset)

[source,bash]
----
$[root@trex]lspci | grep Ethernet
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<1>
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)                #<2>
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) #<3>
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
82:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
82:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
----
<1> Management port 
<2> CIMC port
<3> 10Gb/sec traffic ports (Intel 82599EB) 

=== Obtaining the TRex package

Connect by `ssh` to the TRex machine and execute the commands described below.

NOTE: Prerequisite: *$WEB_URL* is *{web_server_url}* or *{local_web_server_url}* (Cisco internal)

Latest release:
[source,bash]
----
$mkdir trex
$cd trex
$wget --no-cache $WEB_URL/release/latest 
$tar -xzvf latest 
----


Bleeding edge version:
[source,bash]
----
$wget --no-cache $WEB_URL/release/be_latest 
----

To obtain a specific version, do the following: 
[source,bash]
----
$wget --no-cache $WEB_URL/release/vX.XX.tar.gz #<1>
----

<1> X.XX = Version number

=== Running TRex for the first time in loopback 

If you have a 10Gb/sec TRex (based on Intel 520-D2 NICs), you can verify that it works correctly by loopback on the ports.
You can install Intel SFP+ or Cisco SFP+, but you cannot connect ports that are on the same NIC to each other (it might not sync).
If you have only one NIC of 10gb/sec you cannot perform this test because the ports will not have a valid link. 
Another option for loopback is to use link:http://www.fiberopticshare.com/tag/cisco-10g-twinax[Cisco twinax copper cable].

//TBD: perhaps rephase, using a "Prerequisites" or "Required" heading. The requirement here would be: Two (2) 10gb/sec NICs 
//[hh] it is not accurate beacuse with 1Gb/sec you can have this test 

.Correct loopback
image:images/loopback_right.png[title="Correct Loopback"] 

.Wrong loopback
image:images/loopback_wrong.png[title="Wrong Loopback"] 

If you have a 1Gb/Sec Intel NIC (I350) or XL710/X710 NIC, you can do anything you like from the loopback perspective *but* first filter the management port - see xref:trex_config[TRex Configuration].

// above, clarify "you can do anything you like from the loopback perspective"

==== Identify the ports 

[source,bash]
----
 $>sudo ./dpdk_setup_ports.py --s

 Network devices using DPDK-compatible driver
 ============================================

 Network devices using kernel driver
 ===================================
 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active*
 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<1>
 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<2>
 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<3>
 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<4>


 Other network devices
 =====================
 <none>
----
<1> TRex interface #1 before unbinding 
<2> TRex interface #2 before unbinding 
<3> TRex interface #3 before unbinding 
<4> TRex interface #4 before unbinding 

Choose a port to use and follow instructions in the next section to create a configuration file.

==== Create minimum configuration file 

Create a configuration file: `/etc/trex_cfg.yaml`.

You can copy a basic configuration file from cfg folder by running this command...

[source,bash]
----
$cp  cfg/simple_cfg.yaml /etc/trex_cfg.yaml
----

...and edit the configuration file with the desired values.

Example:

[source,bash]
----
<none>
- port_limit    : 4    #<1>     
  version       : 2    #<2>
  interfaces    : ["03:00.0","03:00.1","13:00.1","13:00.0"]  #<3> 
----
<1> Mumber of ports 
<2> Must add version 2 to the configuration file
<3> List of interfaces displayed by `#>sudo ./dpdk_setup_ports.py -s`

When working with a VM, set the destination MAC of one port as the source or the other for loopback the port in the vSwitch 
and you should take the right value from the hypervisor (in case of a physical NIC you can set the MAC address with virtual you can't and you should take it from the hypervisor)
and example

// Clarify paragraph above.

[source,python]
----
        - port_limit      : 2                                                                           
          version       : 2                                                                             
          interfaces    : ["03:00.0","03:00.1"]   <2>                                                      
          port_info       :  # set eh mac addr 
                  - dest_mac        :   [0x1,0x0,0x0,0x1,0x0,0x00]  # port 0                            
                    src_mac         :   [0x2,0x0,0x0,0x2,0x0,0x00]                                     <1>
                  - dest_mac        :   [0x2,0x0,0x0,0x2,0x0,0x00]  # port 1                           <1>
                    src_mac         :   [0x1,0x0,0x0,0x1,0x0,0x00]
----
<1> Source MAC is like destination MAC (this should be set or taken from VMware). The MAC was taken from the hypervisor.
<2> Currently TRex supports only one type of NIC at a time. You cannot mix different NIC types in one config file. For more info, see link:http://trex-tgn.cisco.com/youtrack/issue/trex-197[trex-201].

// where can we describe this limitation (TRex supports only one type of NIC at a time. You cannot mix different NIC types in one config file.) and other limitations?

==== Script for creating config file

We provide a script to create configuration file by specifying as input desired interfaces.
Examples of using:

* Get view of all ports as table:

[source,bash]
----
sudo ./dpdk_setup_ports.py -t
----

* Create config file with given interfaces. You can use PCI addresses or Linux IF (eth1 etc.):

[source,bash]
----
sudo ./dpdk_setup_ports.py -c 13:00.0 eth3 --dump -o /etc/trex_cfg.yaml
### Config file generated by dpdk_setup_ports.py ###

- port_limit: 2
  version: 2
  interfaces: ['13:00.0', '1b:00.0']
  port_info:
      - dest_mac: [0x00, 0x0c, 0x29, 0x2a, 0x99, 0xda] # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  [0x00, 0x0c, 0x29, 0x2a, 0x99, 0xd0]
      - dest_mac: [0x00, 0x0c, 0x29, 0x2a, 0x99, 0xd0] # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  [0x00, 0x0c, 0x29, 0x2a, 0x99, 0xda]

  platform:
      master_thread_id: 0
      latency_thread_id: 1
      dual_if:
        - socket: 0
          threads: [2]
----

.Arguments of dpdk_setup_ports.py script
[options="header",cols="2,5,3",width="100%"]
|=================
| Arg | Description | Example
| -c  | Create a configuration file by specified interfaces (PCI address or Linux names: eth1 etc.) | -c 03:00.1 eth1 eth4 84:00.0
| --dump | Dump created config to screen. |
| -o | Output the config to this file. | -o /etc/trex_cfg.yaml
| --dest-macs | Destination MACs to be used in created yaml file per each interface. Without specifying the option, will be assumed loopback (0&#8660;1, 2&#8660;3 etc.) | --dest-macs 11:11:11:11:11:11 22:22:22:22:22:22
| --ci | Cores include: White list of cores to use. Make sure there is enough for each NUMA. | --ci 0 2 4 5 6
| --ce | Cores exclude: Black list of cores to exclude. Make sure there will be enough for each NUMA. | --ci 10 11 12
| --no-ht | No HyperThreading: Use only one thread of each Core in created config yaml. |
| --prefix | Advanced option: prefix to be used in TRex config in case of parallel instances. | --prefix first_instance
| --zmq-pub-port | Advanced option: ZMQ Publisher port to be used in TRex config in case of parallel instances. | --zmq-pub-port 4000
| --zmq-rpc-port | Advanced option: ZMQ RPC port to be used in TRex config in case of parallel instances. | --zmq-rpc-port
| --ignore-numa | Advanced option: Ignore NUMAs for config creation. Use this option only if you have to, as it might reduce performance. For example, if you have pair of interfaces at different NUMAs |
|=================

==== Run TRex

Use the following command to begin operation of a 4x 10Gb/sec TRex:
[source,bash]
----
$sudo ./t-rex-64 -f cap2/dns.yaml -c 4 -m 1 -d 100  -l 1000 
----

NOTE: For a 10Gb/sec TRex with 2, 6, or 8 ports, add `--limit-ports [number of ports]` *or* follow xref:trex_config[these instructions] to configure TRex. 

If successful, the output will be similar to the following:

[source,python]
----
$ sudo ./t-rex-64 -f cap2/dns.yaml -d 100 -l 1000           
Starting  TRex 1.50 please wait  ... 
zmq publisher at: tcp://*:4500 
 number of ports founded : 4 
  port : 0 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex      <1>
  promiscuous  : 0 
  port : 1 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex
  promiscuous  : 0 
  port : 2 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex
  promiscuous  : 0 
  port : 3 
  ------------
  link         :  link : Link Up - speed 10000 Mbps - full-duplex
  promiscuous  : 0 


 -Per port stats table 
      ports |               0 |               1 |               2 |               3 
 -------------------------------------------------------------------------------------
   opackets |            1003 |            1003 |            1002 |            1002 
     obytes |           66213 |           66229 |           66132 |           66132 
   ipackets |            1003 |            1003 |            1002 |            1002 
     ibytes |           66225 |           66209 |           66132 |           66132 
    ierrors |               0 |               0 |               0 |               0 
    oerrors |               0 |               0 |               0 |               0 
      Tx Bw |     217.09 Kbps |     217.14 Kbps |     216.83 Kbps |     216.83 Kbps 

 -Global stats enabled 
 Cpu Utilization : 0.0  % <12>  29.7 Gb/core <13>
 Platform_factor : 1.0    
 Total-Tx        :     867.89 Kbps                                             <2>
 Total-Rx        :     867.86 Kbps                                             <3>
 Total-PPS       :       1.64 Kpps  
 Total-CPS       :       0.50  cps  

 Expected-PPS    :       2.00  pps   <9>
 Expected-CPS    :       1.00  cps   <10>
 Expected-BPS    :       1.36 Kbps   <11>

 Active-flows    :        0 <6> Clients :      510   Socket-util  : 0.0000 % 
 Open-flows      :        1 <7> Servers :      254   Socket   :        1  Socket/Clients :  0.0  
 drop-rate       :       0.00  bps   <8>
 current time    : 5.3 sec  
 test duration   : 94.7 sec  

 -Latency stats enabled 
 Cpu Utilization : 0.2 %  <14>
 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
 --------------------------------------------------------------------------------------------------
 0 |     1002,    1002,         0,   0,         51  ,      69,       0      |   0  69  67    <4>
 1 |     1002,    1002,         0,   0,         53  ,     196,       0      |   0  196  53   <5>
 2 |     1002,    1002,         0,   0,         54  ,      71,       0      |   0  71  69 
 3 |     1002,    1002,         0,   0,         53  ,     193,       0      |   0  193  52 
----
<1> Link must be up for TRex to work.
<2> Total Rx must be the same as Tx
<3> Total Rx must be the same as Tx 
<4> Tx_ok == Rx_ok
<5> Tx_ok == Rx_ok
<6> Number of TRex active "flows". Could be different than the number of router flows, due to aging issues. Usualy the TRex number of active flows is much lower than that of the router.
<7> Total number of TRex flows opened since startup (including active ones, and ones already closed). 
<8> Drop rate.
<9> Expected number of packets per second (calculated without latency packets).
<10> Expected number of connections per second (calculated without latency packets).
<11> Expected number of bits per second (calculated without latency packets).
<12> Average CPU utilization of transmitters threads. For best results it should be lower than 80%.
<13> Gb/sec generated per core of DP. Higher is better.
<14> Rx and latency thread CPU utilization.


More statistics information:

*socket*::  Same as the active flows.  

*Socket/Clients*:: Average of active flows per client, calculated as active_flows/#clients.

*Socket-util*:: Estimate of how many socket ports are used per client IP. This is approximately ~(100*active_flows/#clients)/64K, calculated as (average active flows per client*100/64K). Utilization of more than 50% means that TRex is generating too many flows per single client, and that more clients must be added.
// clarify above, especially the formula

*Max window*:: Momentary maximum latency for a time window of 500 msec. There are a few numbers per number of windows that are shown. 
 The newest number (last 500msec) is on the right. Oldest in the left. This can help identifying spikes of high latency clearing after some time. Maximum latency is the total maximum over the entire test duration. 
//clarify above

*Platform_factor*:: There are cases in which we duplicate the traffic using splitter/switch and we would like all numbers displayed to be multiplied by this factor (e.g. x2) 
//clarify above

WARNING: If you don't see rx packets, revisit your MAC address configuration.
//clarify above

==== Running TRex for the first time with ESXi:

* Virtual NICs can be used to bridge between TRex and non-supported NICs, or for basic testing. Bandwidth is limited by vSwitch, has IPv6 issues.
// clarify, especially what IPv6 issues

1. Click the host machine, enter Configuration -> Networking.

a. One of the NICs should be connected to the main vSwitch network to get an "outside" connection, for the TRex client and ssh: +
image:images/vSwitch_main.png[title="vSwitch_main"]

b. Other NICs that are used for TRex traffic should be in distinguish vSwitch: +
image:images/vSwitch_loopback.png[title="vSwitch_loopback"]

2. Right-click guest machine -> Edit settings -> Ensure the NICs are set to their networks: +
image:images/vSwitch_networks.png[title="vSwitch_networks"]


[NOTE]
=====================================================================
Current limitation: The following command does not function as expected:
[subs="quotes"]
....
sudo ./t-rex-64 -f cap2/dns.yaml *--lm 1 --lo* -l 1000 -d 100
....
The vSwitch does not "know" where to route the packet. This is expected to be fixed when TRex supports ARP.
=====================================================================

* Pass-through is the way to use directly the NICs from host machine inside the VM. Has no limitations except the NIC/hardware itself. The only difference via bare-metal OS is occasional spikes of latency (~10ms). Passthrough settings cannot be saved to OVA.

1. Click on the host machine. Enter Configuration -> Advanced settings -> Edit. Mark the desired NICs. Reboot the ESXi to apply. +
image:images/passthrough_marking.png[title="passthrough_marking"]

2. Right click on guest machine. Edit settings -> Add -> *PCI device* -> Choose the NICs one by one. +
image:images/passthrough_adding.png[title="passthrough_adding"]

==== Running TRex for the first time with router

You can follow this presentation: link:trex_config_guide.html[first time TRex configuration] 
or continue reading. 
Without config file, TRex sets source MAC of all ports to `00:00:00:01:00:00` and expects to receive packets with this destination MAC address.
So, you just need to configure your router with static ARP entry pointing to the above MAC address.

NOTE: Virtual routers on ESXi (for example, Cisco CSR1000v) must have distinct MAC address for each port. You need to specify the addresses in the configuration file. see more xref:trex_config[here]. Another example is TRex connected to a switch. In this case, each one of the TRex ports should have distinct MAC address.

include::trex_book_basic.asciidoc[]

== Advanced features

=== VLAN Trunk support 

anchor:trex_vlan[]

The VLAN Trunk TRex feature attempts to solve the router port bandwidth limitation when the traffic profile is asymmetric. Example: Asymmetric SFR profile.
This feature converts asymmetric traffic to symmetric, from the port perspective, using router sub-interfaces.
This requires TRex to send the traffic on two VLANs, as described below.

.YAML format 
[source,python]
----
  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
----


.Example
[source,python]
----
- duration : 0.1
  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }   <1>
----
<1> Enable VLAN feature, valn0==100 , valn1==200
        
*Problem definition:*::

Scenario: TRex with two ports and an SFR traffic profile.

.Without VLAN/sub interfaces 
[source,python]
----
0 ( client) -> [  ] - 1 ( server)
----
Without VLAN support the traffic is asymmetric. 10% of the traffic is sent from port 0 (client side), 90% is from port 1 (server). Port 1 become the bottlneck (10Gb/s limit) before port 0.

.With VLAN/sub interfaces 
[source,python]
----
port 0 ( client VLAN0) <->  |  | <-> port 1 ( server-VLAN0)
port 0 ( server VLAN1) <->  |  | <-> port 1 ( client-VLAN1)
----

In this case both ports have the same amount of traffic.

*Router configuation:*::
[source,python]
----
        !
        interface TenGigabitEthernet1/0/0      <1>
         mac-address 0000.0001.0000   
         mtu 4000
         no ip address
         load-interval 30
        !
        i
        interface TenGigabitEthernet1/0/0.100
         encapsulation dot1Q 100               <2> 
         ip address 11.77.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_100_p1_to_p2 <3> 
        !
        interface TenGigabitEthernet1/0/0.200
         encapsulation dot1Q 200               <4>
         ip address 11.88.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_200_p1_to_p2 <5> 
        !
        interface TenGigabitEthernet1/1/0
         mac-address 0000.0001.0000
         mtu 4000
         no ip address
         load-interval 30
        !
        interface TenGigabitEthernet1/1/0.100
         encapsulation dot1Q 100
         ip address 22.77.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_100_p2_to_p1
        !
        interface TenGigabitEthernet1/1/0.200
         encapsulation dot1Q 200
         ip address 22.88.11.1 255.255.255.0
         ip nbar protocol-discovery
         ip policy route-map vlan_200_p2_to_p1
        !
        
        arp 11.77.11.12 0000.0001.0000 ARPA      <6>
        arp 22.77.11.12 0000.0001.0000 ARPA
        
        route-map vlan_100_p1_to_p2 permit 10    <7>
         set ip next-hop 22.77.11.12
        !
        route-map vlan_100_p2_to_p1 permit 10
         set ip next-hop 11.77.11.12
        !
        
        route-map vlan_200_p1_to_p2 permit 10
         set ip next-hop 22.88.11.12
        !
        route-map vlan_200_p2_to_p1 permit 10
         set ip next-hop 11.88.11.12
        !
----
<1> Disable the IP on the main port it is important.
// above, clarify what's important
<2> Enable VLAN1
<3> PBR configuration
<4> Enable VLAN2  
<5> PBR configuration
<6> TRex destination port MAC address
<7> PBR configuration rules

=== Static source MAC address setting  

With this feature, TRex replaces the source MAC address with the client IP address.

 Note: This feature was requested by the Cisco ISG group. 


*YAML:*::
[source,python]
----
 mac_override_by_ip : true
----

.Example
[source,python]
----
- duration : 0.1
 ..
  mac_override_by_ip : true <1>
----
<1> In this case, the client side MAC address looks like this:
SRC_MAC = IPV4(IP) + 00:00  

=== IPv6 support 

Support for IPv6 includes:

1. Support for pcap files containing IPv6 packets
2. Ability to generate IPv6 traffic from pcap files containing IPv4 packets
The following switch enables this feature: `--ipv6`
Two new keywords (`src_ipv6`, `dst_ipv6`) have been added to the YAML file to specify the most significant 96 bits of the IPv6 address - for example:

[source,python]
----
      src_ipv6 : [0xFE80,0x0232,0x1002,0x0051,0x0000,0x0000]
      dst_ipv6 : [0x2001,0x0DB8,0x0003,0x0004,0x0000,0x0000]
----
      
The IPv6 address is formed by placing what would typically be the IPv4
address into the least significant 32-bits and copying the value provided
in the src_ipv6/dst_ipv6 keywords into the most signficant 96-bits.
If src_ipv6 and dst_ipv6 are not specified in the YAML file, the default
is to form IPv4-compatible addresses (where the most signifcant 96-bits
are zero).

There is a support for all plugins (control flows that needed to be changed).
  
*Example:*::
[source,bash]
----
$sudo ./t-rex-64 -f cap2l/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --ipv6
----

*Limitations:*::

* TRex cannot generate both IPv4 and IPv6 traffic. The `--ipv6` switch must be specified even when using a pcap file containing only IPv6 packets.


*Router configuration:*::

[source,python]
----
interface TenGigabitEthernet1/0/0
 mac-address 0000.0001.0000
 mtu 4000
 ip address 11.11.11.11 255.255.255.0
 ip policy route-map p1_to_p2
 load-interval 30
 ipv6 enable   ==> IPv6
 ipv6 address 2001:DB8:1111:2222::1/64                  <1>
 ipv6 policy route-map ipv6_p1_to_p2                    <2>
!


ipv6 unicast-routing                                    <3>

ipv6 neighbor 3001::2 TenGigabitEthernet0/1/0 0000.0002.0002   <4>
ipv6 neighbor 2001::2 TenGigabitEthernet0/0/0 0000.0003.0002

route-map ipv6_p1_to_p2 permit 10                              <5>
 set ipv6 next-hop 2001::2
!
route-map ipv6_p2_to_p1 permit 10
 set ipv6 next-hop 3001::2
!


asr1k(config)#ipv6 route 4000::/64 2001::2                 
asr1k(config)#ipv6 route 5000::/64 3001::2 
----
<1> Enable IPv6 
<2> Add pbr 
<3> Enable IPv6 routing 
<4> MAC address setting should be like TRex
<5> PBR configuraion


=== Source MAC address mapping using a file 

Extends the source MAC address replacment capability. Enables mapping between IPv4->MAC using the `--mac` CLI switch. The file format is YAML.

*Example:*::
[source,bash]
----
$sudo ./t-rex-64 -f cap2/sfr_delay_10_1g.yaml -c 4  -l 100 -d 100000 -m 30  --mac cap2/test_example.yaml
----

*MAC file structure:*::

[source,python]
----
- items :
    - ip : "16.0.0.1"
      mac : [0x16,0x1,0x4,0x5,0x6,0x7]
    - ip : "16.0.0.2"
      mac : [0x16,0x2,0x0,0x1,0x0,0x0]
----

*Limitations:*::

. It is assumed that most clients have a MAC address. At least 90% of IPs should have MAC address mapping.

=== Destination MAC address spreading 

anchor:mac_spread[] 

Using this option, one can send traffic to few destination devices. In normal mode, all packets are sent to the port destination MAC address.
To enable this option, add `--mac-spread` to the command line.

Example:

[source,bash]
----
$sudo ./t-rex-64 -f cap2/http_simple.yaml -d 1000 -m 1000 -c 4 -l 100 --mac-spread 2
----
In this example, TRex sends to port destination MAC and port destination MAC +1. Using a switch, you can connect TRex to multiple devices under test (DUTs). 
All of the DUTs return the traffic only to the correct port source address.
// above, i removed "should" - verify accuracy

[source,bash]
----
        switch A                   switch A
         |                            |
         |   D0+0   -> DUT0  <- D1+0  |
TRex(0) -|                            |-TRex(1)
         |                            |
         |   D0+1  ->  DUT1  <- D1+1  |
                                      |

----


=== NAT support 

TRex can learn dynamic NAT/PAT translation. To enable this feature add `--learn-mode <mode>` to the command line.
To learn the NAT translation, TRex must embed information describing the flow a packet belongs to, in the first
packet of each flow. This can be done in two different methods, depending on the chosen <mode>.

*mode 1:*::

Flow info is embedded in the ACK of the first TCP SYN.
In this mode, there is a limitation that bidirectional UDP templates (for example, DNS) are not supported. 
This mode was developed for testing NAT with firewalls (which usually do not work with mode 2).
In this mode, TRex also learn and compensate for TCP sequence number randomization that might be done by the DUT.
TRex can learn and compensate for seq num randomization in both directions of the connection.

*mode 2:*::

Flow info is added in a special IPv4 option header (8 bytes long 0x10 id). The option is added only to the first packet in the flow.
This mode does not work with DUTs that drop packets with IP options (for example, Cisco ASA firewall).

*mode 3:*::

This is like mode 1, with the only change being that TRex does not learn the seq num randomization in the server->client direction.
This mode can give much better connections per second performance than mode 1 (still, for all existing firewalls, mode 1 cps rate is more than enough).

==== Examples

*simple HTTP traffic*

[source,bash]
----
$sudo ./t-rex-64 -f cap2/http_simple.yaml -c 4  -l 1000 -d 100000 -m 30  --learn-mode 1
----

*SFR traffic without bundeling/ALG support*

[source,bash]
----
$sudo ./t-rex-64 -f avl/sfr_delay_10_1g_no_bundeling.yaml -c 4  -l 1000 -d 100000 -m 10  --learn-mode 2
----

*NAT terminal counters:*::

[source,python]
----
-Global stats enabled 
 Cpu Utilization : 0.6  %  33.4 Gb/core 
 Platform_factor : 1.0
 Total-Tx        :       3.77 Gbps   NAT time out    :      917 <1> (0 in wait for syn+ack) <5>
 Total-Rx        :       3.77 Gbps   NAT aged flow id:        0 <2>
 Total-PPS       :     505.72 Kpps   Total NAT active:      163 <3> (12 waiting for syn) <6>
 Total-CPS       :      13.43 Kcps   Total NAT opened:    82677 <4>
----
<1> Number of connections for which TRex had to send the next packet in the flow, but did not learn the NAT translation yet. Should be 0. Usually, value different than 0 is seen if the DUT drops the flow (probably because it can't handle the number of connections)
<2> Number of flows for which when we got the translation info, flow was aged out already. Non 0 value here should be very rare. Can occur only when there is huge latency in the DUT input/output queue.
<3> Number of flows for which we sent the first packet, but did not learn the NAT translation yet. Value seen depends on the connection per second rate and round trip time.
<4> Total number of translations over the lifetime of the TRex instance. May be different from the total number of flows if template is uni-directional (and consequently does not need translation).
<5> Out of the timed out flows, how many were timed out while waiting to learn the TCP seq num randomization of the server->client from the SYN+ACK packet (Seen only in --learn-mode 1)
<6> Out of the active NAT sessions, how many are waiting to learn the client->server translation from the SYN packet (others are waiting for SYN+ACK from server) (Seen only in --learn-mode 1)

*Configuration for Cisco ASR1000 Series:*::

This feature was tested with the following configuration and  sfr_delay_10_1g_no_bundeling. yaml traffic profile.
Client address range is 16.0.0.1 to 16.0.0.255 

[source,python]
----
interface TenGigabitEthernet1/0/0            <1>
 mac-address 0000.0001.0000
 mtu 4000
 ip address 11.11.11.11 255.255.255.0
 ip policy route-map p1_to_p2
 ip nat inside                               <2>
 load-interval 30
!

interface TenGigabitEthernet1/1/0
 mac-address 0000.0001.0000
 mtu 4000
 ip address 11.11.11.11 255.255.255.0
 ip policy route-map p1_to_p2
 ip nat outside                              <3>
 load-interval 30

ip  nat pool my 200.0.0.0 200.0.0.255 netmask 255.255.255.0  <4>

ip nat inside source list 7 pool my overload 
access-list 7 permit 16.0.0.0 0.0.0.255                      <5>

ip nat inside source list 8 pool my overload                 <6>
access-list 8 permit 17.0.0.0 0.0.0.255                      
----
<1> Must be connected to TRex Client port (router inside port)
<2> NAT inside 
<3> NAT outside
<4> Pool of outside address with overload
<5> Match TRex YAML client range
<6> In case of dual port TRex

// verify 1 and 5 above; rephrased


*Limitations:*::

. The IPv6-IPv6 NAT feature does not exist on routers, so this feature can work on IPv4 only.
. Does not support NAT64. 
. Bundling/plugin support is not fully supported. Consequently, sfr_delay_10.yaml does not work. Use sfr_delay_10_no_bundeling.yaml instead. 
// verify file name "sfr_delay_10_no_bundeling.yaml" above. english spelling is bundling but maybe the filename has the "e"

[NOTE]
=====================================================================
* `--learn-verify` is a TRex debug mechanism for testing the TRex learn mechanism.
* If the router is configured without NAT, it will verify that the inside_ip==outside_ip and inside_port==outside_port.
=====================================================================

=== Flow order/latency verification 

In normal mode (without this feature enabled), received traffic is not checked by software. Hardware (Intel NIC) testin for dropped packets occurs at the end of the test. The only exception is the Latency/Jitter packets. 
This is one reason that with TRex, you *cannot* check features that terminate traffic (for example TCP Proxy).
To enable this feature, add `--rx-check <sample>` to the command line options, where <sample> is the sample rate. 
The number of flows that will be sent to the software for verification is (1/(sample_rate). For 40Gb/sec traffic you can use a sample rate of 1/128. Watch for Rx CPU% utilization.

 INFO: This feature changes the TTL of the sampled flows to 255 and expects to receive packets with TTL 254 or 255 (one routing hop). If you have more than one hop in your setup, use `--hops` to change it to a higher value. More than one hop is possible if there are number of routers betwean TRex client side and TRex server side.

This feature ensures that:

* Packets get out of DUT in order (from each flow perspective).
* There are no packet drops (no need to wait for the end of the test). Without this flag, you must wait for the end of the test in order to identify packet drops, because there is always a difference between TX and Rx, due to RTT.


.Full example 
[source,bash]
----
$sudo ./t-rex-64 -f avl/sfr_delay_10_1g.yaml -c 4 -p -l 100 -d 100000 -m 30  --rx-check 128
----

[source,python]
----
Cpu Utilization : 0.1 %                                                                       <1>
 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
 --------------------------------------------------------------------------------
 0 |     1002,    1002,      2501,   0,         61  ,      70,       3      |  60
 1 |     1002,    1002,      2012,   0,         56  ,      63,       2      |  50
 2 |     1002,    1002,      2322,   0,         66  ,      74,       5      |  68
 3 |     1002,    1002,      1727,   0,         58  ,      68,       2      |  52

 Rx Check stats enabled                                                                       <2>
 -------------------------------------------------------------------------------------------
 rx check:  avg/max/jitter latency,       94  ,     744,       49      |  252  287  309       <3>
 
 active flows: <6>      10, fif: <5>     308,  drop:        0, errors:        0                <4>
 -------------------------------------------------------------------------------------------
----
<1> CPU% of the Rx thread. If it is too high, *increase* the sample rate.
<2> Rx Check section. For more detailed info, press 'r' during the test or at the end of the test.
<3> Average latency, max latency, jitter on the template flows in microseconds. This is usually *higher* than the latency check packet because the feature works more on this packet.
<4> Drop counters and errors counter should be zero. If not, press 'r' to see the full report or view the report at the end of the test.
<5> fif - First in flow. Number of new flows handled by the Rx thread.
<6> active flows - number of active flows handled by rx thread 

.Press R to Display Full Report
[source,python]
----
 m_total_rx                              : 2 
 m_lookup                                : 2 
 m_found                                 : 1 
 m_fif                                   : 1 
 m_add                                   : 1 
 m_remove                                : 1 
 m_active                                : 0 
                                                        <1>
 0  0  0  0  1041  0  0  0  0  0  0  0  0  min_delta  : 10 usec 
 cnt        : 2 
 high_cnt   : 2 
 max_d_time : 1041 usec
 sliding_average    : 1 usec                            <2>
 precent    : 100.0 %
 histogram 
 -----------
 h[1000]  :  2 
 tempate_id_ 0 , errors:       0,  jitter: 61           <3>
 tempate_id_ 1 , errors:       0,  jitter: 0 
 tempate_id_ 2 , errors:       0,  jitter: 0 
 tempate_id_ 3 , errors:       0,  jitter: 0 
 tempate_id_ 4 , errors:       0,  jitter: 0 
 tempate_id_ 5 , errors:       0,  jitter: 0 
 tempate_id_ 6 , errors:       0,  jitter: 0 
 tempate_id_ 7 , errors:       0,  jitter: 0 
 tempate_id_ 8 , errors:       0,  jitter: 0 
 tempate_id_ 9 , errors:       0,  jitter: 0 
 tempate_id_10 , errors:       0,  jitter: 0 
 tempate_id_11 , errors:       0,  jitter: 0 
 tempate_id_12 , errors:       0,  jitter: 0 
 tempate_id_13 , errors:       0,  jitter: 0 
 tempate_id_14 , errors:       0,  jitter: 0 
 tempate_id_15 , errors:       0,  jitter: 0 
 ager :
 m_st_alloc                                 : 1 
 m_st_free                                  : 0 
 m_st_start                                 : 2 
 m_st_stop                                  : 1 
 m_st_handle                                : 0 
----
<1> Errors, if any, shown here
<2> Low pass filter on the active average of latency events 
<3> Error per template info

// IGNORE: this line added to help rendition. Without this line, the "Notes and Limitations" section below does not appear.

*Notes and Limitations:*::

** This feature must be enabled with a latency check (`-l`).
** To receive the packets TRex does the following:
*** Changes the TTL to 0xff and expects 0xFF (loopback) or oxFE (route). (Use `--hop` to configure this value.)
*** Adds 24 bytes of metadata as ipv4/ipv6 option header.
// clarify "ipv4/ipv6 option header" above

== Reference

=== Traffic YAML (parameter of -f option)

==== Global Traffic YAML section 

[source,python]
----
- duration : 10.0                          <1>
  generator :                              <2>
          distribution : "seq"           
          clients_start : "16.0.0.1"     
          clients_end   : "16.0.0.255"   
          servers_start : "48.0.0.1"     
          servers_end   : "48.0.0.255"   
          clients_per_gb : 201
          min_clients    : 101
          dual_port_mask : "1.0.0.0" 
          tcp_aging      : 1
          udp_aging      : 1
  mac        : [0x00,0x00,0x00,0x01,0x00,0x00] <3>
  cap_ipg    : true                            <4>
  cap_ipg_min    : 30                          <5>
  cap_override_ipg    : 200                    <6>
  vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 } <7>
  mac_override_by_ip : true  <8>
----
<1> Test duration (seconds). Can be overridden using the `-d` option.
<2> See the generator section.
// what does note 2 mean? see somewhere else? isn't this simply the generator section?
<3> Default source/destination MAC address. The configuration YAML can override this.
<4> true (default) indicates that the IPG is taken from the cap file (also taking into account cap_ipg_min and cap_override_ipg if they exist). false indicates that IPG is taken from per template section.
<5> The following two options can set the min ipg in microseconds: (if (pkt_ipg<cap_ipg_min) { pkt_ipg=cap_override_ipg} )
<6> Value to override (microseconds), as described in note above.
<7> Enable vlan feature. See xref:trex_vlan[trex_vlan section] for info.
<8> Enable MAC address replacement by client IP.

==== Per template section 
// clarify "per template"  

[source,python]
----
     - name: cap2/dns.pcap <1>
       cps : 10.0          <2>
       ipg : 10000         <3>
       rtt : 10000         <4>
       w   : 1             <5>
       server_addr : "48.0.0.7"    <6>
       one_app_server : true       <7>
       
----
<1> The name of the template pcap file. Can be relative path from the t-rex-64 image directory, or an absolute path. The pcap file should include only one flow. (Exception: in case of plug-ins).
<2> Connection per second. This is the value that will be used if specifying -m 1 from command line (giving -m x will multiply this 
<3> If the global section of the YAML file includes `cap_ipg    : false`, this line sets the inter-packet gap in microseconds. 
<4> Should be set to the same value as ipg (microseconds). 
<5> Default value: w=1. This indicates to the IP generator how to generate the flows. If w=2, two flows from the same template will be generated in a burst (more for HTTP that has burst of flows).
<6> If `one_app_server` is set to true, then all templates will use the same server.
<7> If the same server address is required, set this value to true.



=== Configuration YAML (parameter of --cfg option)

anchor:trex_config[]

The configuration file, in YAML format, configures TRex behavior, including:

 
- MAC address for each port (source and destination)
- Masking interfaces (usually for 1Gb/Sec TRex) to ensure that TRex does not take the management ports as traffic ports.
- Changing the zmq/telnet TCP port.

==== Basic Configuration 

Copy/install the configuration file to `/etc/trex_cfg.yaml`. 
TRex loads it automatically at startup. You still can override options with the command line option switch `--cfg [file]` in the CLI
Configuration file examples can be found in the `$ROOT/cfg` folder


[source,python]
----
        - port_limit    : 2    #mandatory                                                               <1>
          version       : 2    #mandatory                                                               <2>
          interfaces    : ["03:00.0","03:00.1"]   #mandatory                                            <3>          
          #enable_zmq_pub  : true                                                                       <4>
          #zmq_pub_port    : 4500                                                                       <5>
          #prefix          : setup1                                                                     <6>
          #limit_memory    : 1024                                                                       <7>
          c               : 4                                                                           <8>
          port_bandwidth_gb : 10                                                                        <9>
          port_info       :  # set eh mac addr  mandatory
                  - dest_mac        :   [0x1,0x0,0x0,0x1,0x0,0x00]  # port 0                            <10>
                    src_mac         :   [0x2,0x0,0x0,0x2,0x0,0x00]
                  - dest_mac        :   [0x3,0x0,0x0,0x3,0x0,0x00]  # port 1
                    src_mac         :   [0x4,0x0,0x0,0x4,0x0,0x00]
                  - dest_mac        :   [0x5,0x0,0x0,0x5,0x0,0x00]  # port 2
                    src_mac         :   [0x6,0x0,0x0,0x6,0x0,0x00]
                  - dest_mac        :   [0x7,0x0,0x0,0x7,0x0,0x01]  # port 3
                    src_mac         :   [0x0,0x0,0x0,0x8,0x0,0x02]
                  - dest_mac        :   [0x0,0x0,0x0,0x9,0x0,0x03]  # port 4
----
<1>  The number of ports, should be equal to the number of interfaces in 3)   - mandatory
<2>  Must be set to 2                                                         - mandatory
<3>  Interface that should be used. used `sudo ./dpdk_setup_ports.py --show`  - mandatory
<4>  Enable the ZMQ publisher for stats data, default is true. 
<5>  ZMQ port number. the default value is good. you can remove this line 

<6> The name of the setup should be distinct ( DPDK --file-prefix )
<7> DPDK -m limit the packet memory 
<8> Number of threads per dual interface ( like -c CLI option )
<9> The bandwidth of each interface in Gb/sec. In this example we have 10Gb/sec interfaces. for VM put 1. it used to tune the amount of memory allocated by TRex.
<10>  MAC address per port - source and destination.


To find out what the interfaces ids, perform the following:

[source,bash]
----
 $>sudo ./dpdk_setup_ports.py --show

 Network devices using DPDK-compatible driver
 ============================================

 Network devices using kernel driver
 ===================================
 0000:02:00.0 '82545EM Gigabit Ethernet Controller (Copper)' if=eth2 drv=e1000 unused=igb_uio *Active*
 0000:03:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<1>
 0000:03:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<2>
 0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<3>
 0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv= unused=ixgb #<4>

 Other network devices
 =====================
 <none>
----
<1> TRex interface #1 before unbinding 
<2> TRex interface #2 before unbinding 
<3> TRex interface #3 before unbinding 
<4> TRex interface #4 before unbinding 


minimum configuration file is:

[source,bash]
----
<none>
- port_limit    : 4         
  version       : 2   #<1>
  interfaces    : ["03:00.0","03:00.1","13:00.1","13:00.0"]  #<2> 
----
<1> must add version 2 to the configuration file
<2> The list of interfaces from `sudo ./dpdk_setup_ports.py --show`


==== Memory section configuration  

The memory section is optional. It is used when there is a need to tune the amount of memory used by packet manager 

[source,python]
----
        - port_limit      : 2                                                                           
          version       : 2                                                                             
          interfaces    : ["03:00.0","03:00.1"]                                                         
          memory    :                                           <1>
             mbuf_64     : 16380                                <2>
             mbuf_128    : 8190
             mbuf_256    : 8190
             mbuf_512    : 8190
             mbuf_1024   : 8190
             mbuf_2048   : 4096
             traffic_mbuf_64     : 16380                        <3>
             traffic_mbuf_128    : 8190
             traffic_mbuf_256    : 8190
             traffic_mbuf_512    : 8190
             traffic_mbuf_1024   : 8190
             traffic_mbuf_2048   : 4096
             dp_flows    : 1048576                              <4>
             global_flows : 10240                               <5>
----
<1> Memory section 
<2> Per dual interfaces number of buffers - buffer for real time traffic generation 
<3> Traffic buffer - when you have many template only this section should be enlarge  
<4> number of TRex flows needed 
<5> reserved 


==== Platform section configuration  

The platform section is optional. It is used to tune the performance and allocate the cores to the right NUMA  
a configuration file now has the folowing struct to support multi instance 

[source,python]
----
- version       : 2
  interfaces    : ["03:00.0","03:00.1"]                         
  port_limit    : 2 
  prefix          : setup1                                      <1>
  limit_memory    : 1024                                        <2>
  c               : 4                                           <3>
  port_bandwidth_gb : 10                                        <4>
  platform :                                                    <5>
        master_thread_id  : 0                                   <6>        
        latency_thread_id : 5                                   <7>
        dual_if   :                                             
             - socket   : 0                                     <8>
               threads  : [1,2,3,4]                             <9>
----
<1> The name of the setup should be distinct ( DPDK --file-prefix )
<2> DPDK -m 
<3> Number of threads per dual interface ( like -c CLI option )
<4> The bandwidth of each interface in Gb/sec. In this example we have 10Gb/sec interfaces. for VM put 1. it used to tune the amount of memory allocated by TRex.
<5> the platform section
<6> The thread_id for control 
<7> The thread_id for latency if used
<8> Socket of the dual interfaces, in this example of 03:00.0 and 03:00.1, memory should be local to the interface. (Currently dual interface can't use 2 NUMAs.)
<9> Thread to be used, should be local to the NIC. The threads are pinned to cores, thus specifying threads is like specifying cores.

*Real example:* anchor:numa-example[]

We've connected 2 Intel XL710 NICs close to each other on motherboard, they shared same NUMA:

image:images/same_numa.png[title="2_NICSs_same_NUMA"]

The CPU utilization was very high ~100%, with c=2 and c=4 the results were same.

Then, we moved the cards to different NUMAs:

image:images/different_numa.png[title="2_NICSs_different_NUMAs"]

*+*
We added configuration to the /etc/trex_cfg.yaml:

[source,python]
  platform :
        master_thread_id  : 0
        latency_thread_id : 8
        dual_if   :
             - socket   : 0
               threads  : [1, 2, 3, 4, 5, 6, 7]
             - socket   : 1
               threads  : [9, 10, 11, 12, 13, 14, 15]

This gave best results: with *\~98 Gb/s* TX BW and c=7, CPU utilization became *~21%*! (40% with c=4)

=== Command line options 

anchor:cml-line[]

*-f=TRAFIC_YAML_FILE*:: 
       Traffic YAML configuration file.

*-c=CORES*::
  Number of cores _per dual interface_. Use 4 for TRex 40Gb/sec. Monitor the CPU% of TRex - it should be ~50%. +
  TRex uses 2 cores for inner needs, the rest of cores can be used divided by number of dual interfaces. +
  For virtual NICs the limit is -c=1.

*-l=HZ*::
    Run the latency daemon in this Hz rate. Example: -l 1000 runs 1000 pkt/sec from each interface. A value of zero (0) disables the latency check.

*-d=DURATION*::
        Duration of the test (sec), Default: 0 

*-m=MUL*::
   Factor for bandwidth (multiply the CPS of each template by this value).

*--ipv6*::
       Convert template to IPv6 mode.

*--learn-mode <mode (1-2)>*::
    Learn the dynamic NAT translation. +
    1 - Use TCP ACK in first SYN to pass NAT translation information. Will work only for TCP streams. Initial SYN packet must be present in stream. +
    2 - Add special IP option to pass NAT translation information. Will not work on certain firewalls if they drop packets with IP options.

*--learn-verify*::
    Learn the translation. This feature is intended for verification of the mechanism in cases where there is no NAT.
  
*-p*::                
  Flow-flip. Sends all flow packets from the same interface. This can solve the flow order. Does not work with any router configuration.

*-e*::                
  same as `-p` but comply to the direction rules and replace source/destination IPs. it might not be good for NBAR as it is expected clients ip to be sent from same direction.

//TBD: The last 2 sentences (flow order, router configuration) are unclear.

  
*--lm=MASK*::                        
  Latency mask. Use this to verify port connectivity. Possible values: 0x1 (only port 0 will send traffic), 0x2 (only port 1 will send traffic). 

*--lo*::                         
   Latency test.
  
*--limit-ports=PORTS*::
   Limit number of ports. Configure this in the --cfg file. Possible values (number of ports): 2, 4, 6, 8. (Default: 4)
  
*--nc*::       
    If set, will terminate exacly at the end of the duration. This provides a faster, more accurate TRex termination. In default it wait for all the flow to terminate gracefully. In case of a very long flow the termination might be prolong. 
    
*-pm=MULTIFLIER*::
   Platform factor. If the setup includes a splitter, you can multiply the total results by this factor. Example: --pm 2.0 will multiply all bps results by this factor.
  
*-pubd*::
  Disable ZMQ monitor's  publishers.

*-1g*::                       
   Deprecated. Configure TRex to 1G. Configure this in the --cfg file. 
  
*-k=SEC*::
   Run a latency test before starting the test. TRex will wait for x sec before and after sending latency packets at startup.
   Current limitation (holds for TRex version 1.82): does not work properly on VM.

*-w=SEC*::
   Wait additional time between NICs initialization and sending traffic. Can be useful if DUT needs extra setup time. Default is 1 second.
  
*--cfg=platform_yaml*::     
        Load and configure platform using this file. See example file: cfg/cfg_examplexx.yaml 
        This file is used to configure/mask interfaces, cores, affinity, and MAC addresses.  
        You can use the example file by copying it to: /etc/trex_cfg.yaml   
  
  
*-v=VERBOSE*::
   Verbose mode (works only on the debug image! )  
      1    Show only stats.
      2    Run preview. Does not write to file.
      3    Run preview and write to stats file.
  Note: When using verbose mode, it is not necessary to add an output file. 
  Caution: Operating in verbose mode can generate very large files (terabytes). Use with caution, only on a local drive.
 
  
*--rx-check=SAMPLE_RATE*::
        Enable Rx check module. Using this each thread samples flows (1/sample) and checks order, latency, and additional statistics.
        Note: This feature operates as an additional thread.
  
*--hops=HOPES*::
   Number of hops in the setup (default is one hop). Relevant only if the Rx check is enabled.
   
*--iom=MODE*::
        I/O mode for interactive mode. Possible values: 0 (silent), 1 (normal), 2 (short)
  
*--no-flow-control-change*::
  Prevents TRex from changing flow control. By default (without this option), TRex disables flow control at startup for all cards, except for the Intel XL710 40G card.

*--mac-spread*::            
  Spread the destination mac by this this factor. e.g 2 will generate the traffic to 2 devices DEST-MAC ,DEST-MAC+1. The maximum is up to 128 devices.


ifndef::backend-docbook[]


endif::backend-docbook[]

== Appendix

=== Simulator 
 
The TRex simulator is a linux application that can process on any Linux CEL (it can run on TRex itself).
you can create create output pcap file from input of traffic YAML.

====  Simulator 


[source,bash]
----

$./bp-sim-64-debug -f avl/sfr_delay_10_1g.yaml -v 1

 -- loading cap file avl/delay_10_http_get_0.pcap 
 -- loading cap file avl/delay_10_http_post_0.pcap 
 -- loading cap file avl/delay_10_https_0.pcap 
 -- loading cap file avl/delay_10_http_browsing_0.pcap 
 -- loading cap file avl/delay_10_exchange_0.pcap 
 -- loading cap file avl/delay_10_mail_pop_0.pcap 
 -- loading cap file avl/delay_10_mail_pop_1.pcap 
 -- loading cap file avl/delay_10_mail_pop_2.pcap 
 -- loading cap file avl/delay_10_oracle_0.pcap 
 -- loading cap file avl/delay_10_rtp_160k_full.pcap 
 -- loading cap file avl/delay_10_rtp_250k_full.pcap 
 -- loading cap file avl/delay_10_smtp_0.pcap 
 -- loading cap file avl/delay_10_smtp_1.pcap 
 -- loading cap file avl/delay_10_smtp_2.pcap 
 -- loading cap file avl/delay_10_video_call_0.pcap 
 -- loading cap file avl/delay_10_sip_video_call_full.pcap 
 -- loading cap file avl/delay_10_citrix_0.pcap 
 -- loading cap file avl/delay_10_dns_0.pcap 
 id,name                                    , tps, cps,f-pkts,f-bytes, duration,   Mb/sec,   MB/sec,   c-flows,  PPS,total-Mbytes-duration,errors,flows    #<2>
 00, avl/delay_10_http_get_0.pcap             ,404.52,404.52,    44 ,   37830 ,   0.17 , 122.42 ,   15.30 ,         67 , 17799 ,       2 , 0 , 1 
 01, avl/delay_10_http_post_0.pcap            ,404.52,404.52,    54 ,   48468 ,   0.21 , 156.85 ,   19.61 ,         85 , 21844 ,       2 , 0 , 1 
 02, avl/delay_10_https_0.pcap                ,130.87,130.87,    96 ,   91619 ,   0.22 ,  95.92 ,   11.99 ,         29 , 12564 ,       1 , 0 , 1 
 03, avl/delay_10_http_browsing_0.pcap        ,709.89,709.89,    37 ,   34425 ,   0.13 , 195.50 ,   24.44 ,         94 , 26266 ,       2 , 0 , 1 
 04, avl/delay_10_exchange_0.pcap             ,253.81,253.81,    43 ,    9848 ,   1.57 ,  20.00 ,    2.50 ,        400 , 10914 ,       0 , 0 , 1 
 05, avl/delay_10_mail_pop_0.pcap             ,4.76,4.76,    20 ,    5603 ,   0.17 ,   0.21 ,    0.03 ,          1 ,    95 ,       0 , 0 , 1 
 06, avl/delay_10_mail_pop_1.pcap             ,4.76,4.76,   114 ,  101517 ,   0.25 ,   3.86 ,    0.48 ,          1 ,   543 ,       0 , 0 , 1 
 07, avl/delay_10_mail_pop_2.pcap             ,4.76,4.76,    30 ,   15630 ,   0.19 ,   0.60 ,    0.07 ,          1 ,   143 ,       0 , 0 , 1 
 08, avl/delay_10_oracle_0.pcap               ,79.32,79.32,   302 ,   56131 ,   6.86 ,  35.62 ,    4.45 ,        544 , 23954 ,       0 , 0 , 1 
 09, avl/delay_10_rtp_160k_full.pcap          ,2.78,8.33,  1354 , 1232757 ,  61.24 ,  27.38 ,    3.42 ,        170 ,  3759 ,       0 , 0 , 3 
 10, avl/delay_10_rtp_250k_full.pcap          ,1.98,5.95,  2069 , 1922000 ,  61.38 ,  30.48 ,    3.81 ,        122 ,  4101 ,       0 , 0 , 3 
 11, avl/delay_10_smtp_0.pcap                 ,7.34,7.34,    22 ,    5618 ,   0.19 ,   0.33 ,    0.04 ,          1 ,   161 ,       0 , 0 , 1 
 12, avl/delay_10_smtp_1.pcap                 ,7.34,7.34,    35 ,   18344 ,   0.21 ,   1.08 ,    0.13 ,          2 ,   257 ,       0 , 0 , 1 
 13, avl/delay_10_smtp_2.pcap                 ,7.34,7.34,   110 ,   96544 ,   0.27 ,   5.67 ,    0.71 ,          2 ,   807 ,       0 , 0 , 1 
 14, avl/delay_10_video_call_0.pcap           ,11.90,11.90,  2325 , 2532577 ,  36.56 , 241.05 ,   30.13 ,        435 , 27662 ,       3 , 0 , 1 
 15, avl/delay_10_sip_video_call_full.pcap    ,29.35,58.69,  1651 ,  120315 ,  24.56 ,  28.25 ,    3.53 ,        721 , 48452 ,       0 , 0 , 2 
 16, avl/delay_10_citrix_0.pcap               ,43.62,43.62,   272 ,   84553 ,   6.23 ,  29.51 ,    3.69 ,        272 , 11866 ,       0 , 0 , 1 
 17, avl/delay_10_dns_0.pcap                  ,1975.02,1975.02,     2 ,     162 ,   0.01 ,   2.56 ,    0.32 ,         22 ,  3950 ,       0 , 0 , 1 

 00, sum                                      ,4083.86,93928.84,  8580 , 6413941 ,   0.00 , 997.28 ,  124.66 ,       2966 , 215136 ,      12 , 0 , 23 
 Memory usage 
 size_64        : 1687 
 size_128       : 222 
 size_256       : 798 
 size_512       : 1028 
 size_1024      : 86 
 size_2048      : 4086 
 Total    :       8.89 Mbytes  159% util #<1>

----
<1> the memory usage of the templates 
<2> CSV for all the templates 


=== firmware update to XL710/X710 
anchor:xl710-firmware[]
 
To upgrade the firmware  follow  this

==== Download the driver 

*Download driver i40e from link:https://downloadcenter.intel.com/download/24411/Network-Adapter-Driver-for-PCI-E-40-Gigabit-Network-Connections-under-Linux-[here]
*Build the kernel module

[source,bash]
----
$tar -xvzf i40e-1.3.47
$cd i40e-1.3.47/src
$make 
$sudo insmod  i40e.ko
----


====  Bind the NIC to Linux

In this stage we bind the NIC to Linux (take it from DPDK)

[source,bash]
----
$sudo ./dpdk_nic_bind.py --status # show the ports 

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 'Device 1583' drv=igb_uio unused=      #<1>
0000:02:00.1 'Device 1583' drv=igb_uio unused=      #<2>
0000:87:00.0 'Device 1583' drv=igb_uio unused=
0000:87:00.1 'Device 1583' drv=igb_uio unused=

$sudo dpdk_nic_bind.py -u 02:00.0  02:00.1          #<3> 

$sudo dpdk_nic_bind.py -b i40e 02:00.0 02:00.1      #<4>

$ethtool -i p1p2                                    #<5>  

driver: i40e
version: 1.3.47
firmware-version: 4.24 0x800013fc 0.0.0             #<6>
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

   
$ethtool -S p1p2 
$lspci -s 02:00.0 -vvv                              #<7>


----
<1> XL710 ports that need to unbind from DPDK
<2> XL710 ports that need to unbind from DPDK
<3> Unbind from DPDK using this command
<4> Bind to linux to i40e driver 
<5> Show firmware version throw linux driver 
<6> Firmare version
<7> More info 


====  Upgrade 

Download NVMUpdatePackage.zip from Intel site link:http://downloadcenter.intel.com/download/24769/NVM-Update-Utility-for-Intel-Ethernet-Converged-Network-Adapter-XL710-X710-Series[here]  
It includes the utility `nvmupdate64e`

Run this:

[source,bash]
----
$sudo ./nvmupdate64e  
----

You might need a power cycle and to run this command a few times to get the latest firmware  

====  QSFP+ support for XL710

see link:https://www.google.co.il/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjJhPSH3b3LAhUp7nIKHSkACUYQFggaMAA&url=http%3A%2F%2Fwww.intel.co.id%2Fcontent%2Fdam%2Fwww%2Fpublic%2Fus%2Fen%2Fdocuments%2Frelease-notes%2Fxl710-ethernet-controller-feature-matrix.pdf&usg=AFQjCNFhwozfz-XuKGMOy9_MJDbetw15Og&sig2=ce7YU9F9Et6xf6KvqSFBxg&bvm=bv.116636494,d.bGs[QSFP+ support] for QSFP+ support and Firmware requirement for XL710


=== TRex with ASA 5585

When running TRex aginst ASA 5585, you have to notice following things:

* ASA can't forward ipv4 options, so there is a need to use --learn-mode 1 (or 3) in case of NAT. In this mode, bidirectional UDP flows are not supported.
--learn-mode 1 support TCP sequence number randomization in both sides of the connection (client to server and server client). For this to work, TRex must learn
the translation of packets from both sides, so this mode reduce the amount of connections per second TRex can generate (The number is still high enough to test
any existing firewall). If you need higher cps rate, you can use --learn-mode 3. This mode handles sequence number randomization on client->server side only.
* Latency should be tested using ICMP with `--l-pkt-mode 2`

====  ASA 5585 sample configuration

[source,bash]
----
ciscoasa# show running-config 
: Saved

: 
: Serial Number: JAD194801KX
: Hardware:   ASA5585-SSP-10, 6144 MB RAM, CPU Xeon 5500 series 2000 MHz, 1 CPU (4 cores)
:
ASA Version 9.5(2) 
!
hostname ciscoasa
enable password 8Ry2YjIyt7RRXU24 encrypted
passwd 2KFQnbNIdI.2KYOU encrypted
names
!
interface Management0/0
 management-only
 nameif management
 security-level 100
 ip address 10.56.216.106 255.255.255.0 
!             
interface TenGigabitEthernet0/8
 nameif inside
 security-level 100
 ip address 15.0.0.1 255.255.255.0 
!             
interface TenGigabitEthernet0/9
 nameif outside
 security-level 0
 ip address 40.0.0.1 255.255.255.0 
!             
boot system disk0:/asa952-smp-k8.bin
ftp mode passive
pager lines 24
logging asdm informational
mtu management 1500
mtu inside 9000
mtu outside 9000
no failover   
no monitor-interface service-module 
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp outside 40.0.0.2 90e2.baae.87d1 
arp inside 15.0.0.2 90e2.baae.87d0 
arp timeout 14400
no arp permit-nonconnected
route management 0.0.0.0 0.0.0.0 10.56.216.1 1
route inside 16.0.0.0 255.0.0.0 15.0.0.2 1
route outside 48.0.0.0 255.0.0.0 40.0.0.2 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
user-identity default-domain LOCAL
http server enable
http 192.168.1.0 255.255.255.0 management
no snmp-server location
no snmp-server contact
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet 0.0.0.0 0.0.0.0 management
telnet timeout 5
ssh stricthostkeycheck
ssh timeout 5 
ssh key-exchange group dh-group1-sha1
console timeout 0
!             
tls-proxy maximum-session 1000
!             
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
dynamic-access-policy-record DfltAccessPolicy
!             
class-map icmp-class
 match default-inspection-traffic
class-map inspection_default
 match default-inspection-traffic
!             
!             
policy-map type inspect dns preset_dns_map
 parameters   
  message-length maximum client auto
  message-length maximum 512
policy-map icmp_policy
 class icmp-class
  inspect icmp 
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect ip-options 
!             
service-policy global_policy global
service-policy icmp_policy interface outside
prompt hostname context 
!             
jumbo-frame reservation
!             
no call-home reporting anonymous
: end         
ciscoasa# 
----

====  TRex commands example 

Using these commands the configuration is:

1. NAT learn mode (TCP-ACK)
2. Delay of 1 second at start up (-k 1). It was added because ASA drops the first packets.
3. Latency is configured to ICMP reply mode (--l-pkt-mode 2).


*Simple HTTP:*::
[source,bash]
----
$sudo ./t-rex-64 -f cap2/http_simple.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 1000  --learn-mode 1 -k 1
----

This is more realistic traffic for enterprise (we removed from SFR file the bidirectional UDP traffic templates, which (as described above), are not supported in this mode).

*Enterprise profile:*::
[source,bash]
----
$sudo ./t-rex-64 -f avl/sfr_delay_10_1g_asa_nat.yaml -d 1000 -l 1000 --l-pkt-mode 2 -m 4 --learn-mode 1 -k 1
----

The TRex output

[source,bash]
----
-Per port stats table 
      ports |               0 |               1 
 -----------------------------------------------------------------------------------------
   opackets |       106347896 |       118369678 
     obytes |     33508291818 |    118433748567 
   ipackets |       118378757 |       106338782 
     ibytes |    118434305375 |     33507698915 
    ierrors |               0 |               0 
    oerrors |               0 |               0 
      Tx Bw |     656.26 Mbps |       2.27 Gbps 

-Global stats enabled 
 Cpu Utilization : 18.4  %  31.7 Gb/core 
 Platform_factor : 1.0  
 Total-Tx        :       2.92 Gbps   NAT time out    :        0 #<1> (0 in wait for syn+ack) #<1>
 Total-Rx        :       2.92 Gbps   NAT aged flow id:        0 #<1>
 Total-PPS       :     542.29 Kpps   Total NAT active:      163  (12 waiting for syn)
 Total-CPS       :       8.30 Kcps   Nat_learn_errors:        0

 Expected-PPS    :     539.85 Kpps
 Expected-CPS    :       8.29 Kcps  
 Expected-BPS    :       2.90 Gbps  

 Active-flows    :     7860  Clients :      255   Socket-util : 0.0489 %    
 Open-flows      :  3481234  Servers :     5375   Socket :     7860 Socket/Clients :  30.8 
 drop-rate       :       0.00  bps   #<1>
 current time    : 425.1 sec  
 test duration   : 574.9 sec  

-Latency stats enabled 
 Cpu Utilization : 0.3 %  
 if|   tx_ok , rx_ok  , rx   ,error,    average   ,   max         , Jitter ,  max window 
   |         ,        , check,     , latency(usec),latency (usec) ,(usec)  ,             
 ---------------------------------------------------------------------------------------------------------------- 
 0 |   420510,  420495,         0,   1,         58  ,    1555,      14      |  240  257  258  258  219  930  732  896  830  472  190  207  729 
 1 |   420496,  420509,         0,   1,         51  ,    1551,      13      |  234  253  257  258  214  926  727  893  826  468  187  204  724
----
<1>  These counters should be zero 

anchor:fedora21_example[]

=== Fedora 21 Server installation

Download the .iso file from link above, boot with it using Hypervisor or CIMC console. +
Troubleshooting -> install in basic graphics mode

* In packages selection, choose:

** C Development Tools and Libraries

** Development Tools

** System Tools

* Set Ethernet configuration if needed

* Use default hard-drive partitions, reclaim space if needed

* After installation, edit file /etc/selinux/config +
set: +
SELINUX=disabled

* Run: +
systemctl disable firewalld 

* Edit file /etc/yum.repos.d/fedora-updates.repo +
set everywhere: +
enabled=0

* Reboot

=== Configure Linux host as network emulator

There are lots of Linux tutorials on the web, so this will not be full tutorial, only highlighting some key points. Commands
were checked on Ubuntu system.

For this example:

1. TRex Client side network is 16.0.0.x 
2. TRex Server side network is 48.0.0.x 
3. Linux Client side network eth0 is configured with IPv4 as 172.168.0.1 
4. Linux Server side network eth1 is configured with IPv4 as 10.0.0.1 

[source,bash]
----

  TRex-0 (16.0.0.1->48.0.0.1 )   <-->

                ( 172.168.0.1/255.255.0.0)-eth0 [linux] -( 10.0.0.1/255.255.0.0)-eth1 

                <--> TRex-1 (16.0.0.1<-48.0.0.1)
  
----


==== Enable forwarding
One time (will be discarded after reboot): +

[source,bash]
----
echo 1 > /proc/sys/net/ipv4/ip_forward
----
To make this permanent, add the following line to the file /etc/sysctl.conf: +
----
net.ipv4.ip_forward=1
----

==== Add static routes
Example if for the default TRex networks, 48.0.0.0 and 16.0.0.0.

Routing all traffic from 48.0.0.0 to the gateway 10.0.0.100
[source,bash]
----
route add -net 48.0.0.0 netmask 255.255.0.0 gw 10.0.0.100
----

Routing all traffic from 16.0.0.0 to the gateway 172.168.0.100
[source,bash]
----
route add -net 16.0.0.0 netmask 255.255.0.0 gw 172.168.0.100
----
If you use stateless mode, and decide to add route only in one direction, remember to disable reverse path check. +
For example, to disable on all interfaces:
[source,bash]
----
for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
  echo 0 > $i 
done
----

Alternatively, you can edit /etc/network/interfaces, and add something like this for both ports connected to TRex.
This will take effect, only after restarting networking (rebooting the machine in an alternative also).
----
auto eth1
iface eth1 inet static
address 16.0.0.100
netmask 255.0.0.0
network 16.0.0.0
broadcast 16.255.255.255
... same for 48.0.0.0
----

==== Add static ARP entries
[source,bash]
----
sudo arp -s 10.0.0.100 <Second TRex port MAC>
sudo arp -s 172.168.0.100 <TRex side the NICs are not visible to ifconfig, run:
----