path: root/docs/test_specs/csit-nat44ed-cps-spec.md

## Content

<!-- MarkdownTOC autolink="true" -->

- [Tests for NAT44ED](#tests-for-nat44ed)
- [CPS Test Objectives](#cps-test-objectives)
- [Input Parameters](#input-parameters)
- [Stateful traffic profiles](#stateful-traffic-profiles)
- [UDP CPS Tests](#udp-cps-tests)
  - [UDP TRex Measurements](#udp-trex-measurements)
    - [Counters](#counters)
    - [Calculations](#calculations)
    - [CPS-MRR](#cps-mrr)
    - [CPS-PDR](#cps-pdr)
    - [CPS-NDR](#cps-ndr)
  - [UDP VPP Telemetry](#udp-vpp-telemetry)
    - [Counters](#counters-1)
    - [Errors](#errors)
- [TCP/IP CPS Tests](#tcpip-cps-tests)
  - [TCP/IP TRex Measurements](#tcpip-trex-measurements)
    - [Counters](#counters-2)
    - [Calculations](#calculations-1)
    - [CPS Trial PASS](#cps-trial-pass)
    - [CPS-MRR](#cps-mrr-1)
    - [CPS-PDR](#cps-pdr-1)
    - [CPS-NDR](#cps-ndr-1)
  - [TCP/IP VPP Telemetry](#tcpip-vpp-telemetry)
    - [Counters](#counters-3)
    - [Errors](#errors-1)

<!-- /MarkdownTOC -->

## Tests for NAT44ED

Two types of stateful tests are developed for NAT44ED (source network address
and port translation IPv4 to IPv4 with 5-tuple session state):

- Connections-Per-Second (CPS), discovering the maximum rate of creating
  NAT44ED sessions. Measured separately for UDP and TCP connections and
  for different session scale.

- Packets-Per-Second (PPS), discovering the maximum rate of
simultaneously creating NAT44ED sessions and transfering bulk of data
packets across the corresponding connections. Measured separately for
UDP and TCP connections with different session scale and different data
packet sizes per each connection. Current code is using 64B only for UDP
and default MSS 1460B for TCP/IP.

This note describes CPS tests.

## CPS Test Objectives

Discover DUT's highest sustain rate of creating fully functional NAT44ED
5-tuple stateful session entries. Session entry is considered fully
functional, if packets associated with this entry are NAT44ED processed
by DUT and forwarded in both directions without loss.

Similarly to packet throughput tests, three CPS rates are discovered:

- CPS-MRR, verified connection rate at maximal connection attempt rate,
  regardless of an amount of not established connections. (Connections
  per Second - Maximum Receive Rate.)
- CPS-NDR, maximal connection attempt rate at which all connections get
  established. (Connections per Second - Non Drop Rate.)
- CPS-PDR, maximal connection attempt rate at which ratio of not
  established connections to attempted connections is below configured
  threshold. (Connections per Second - Partial Drop Rate.)

## Input Parameters

- `max_cps_rate`, maximum rate of attempting connections, to be used by
  traffic generator, limited by traffic generator capabilities, Ethernet
  link(s) rate and NIC model.
- `min_cps_rate`, minimum rate of establishing connections to be used for
  measurements. Search fails if lower transmit rate needs to be used to
  meet search criteria.
- `target_session_number`, maximum number of sessions to be established and
  tested.
- `target_loss_ratio`, maximum acceptable connections loss ratio search
  criteria for PDR measurements with UDP tests. Indicates packet drop
  impact on connection establishment rate.
- `final_relative_width`, required measurement resolution expressed as
  (lower_bound, upper_bound) interval width relative to upper_bound.
- stateful traffic profiles, TRex ASTF program defining the connection
  per L4 protocol tested (TCP, UDP), including connect and
  close sequence.

## Stateful traffic profiles

TRex ASTF program defines following TCP and UDP transactions for
discovering NAT44ED CPS limits:

- CPS with TCP
  - connect(syn,syn-ack,ack)
    - pkts client tx 2, rx 1
    - pkts server tx 1, rx 2
  - delay (note: optional, currently not implemented)
    - no packets
  - close(fin,fin-ack,ack,ack)
    - pkts client tx 2, rx 2
    - pkts server tx 1, rx 2
- CPS with UDP
  - connect_and_close(req,ack)
    - pkts client tx 1, rx 1
    - pkts server tx 1, rx 1

TRex ASTF program configuration parameters:

- `limit` of connections, set to `target_session_number`.
- `multiplier`, represents `trial_cps_rate`, a number of connections per
  second to be executed per trial. Multiplier applies to connect phases.
  Close phases occur automatically based on arrival of the last packet
  expected per session.
- IPv4 source and destination address and port ranges matching the
  limit of connections.
  - Source and destination addresses changing packet-by-packet with two
    separate profiles i) incrementing sequentially pair-wise
    (implemented) and ii) changed randomly (with seed) pair-wise (not
    implemented yet).
  - Source port changing randomly within the range.
- `trial_duration`, function of `target_session_number` and `multiplier`
  - `multiplier`, subject of the search, value in the range (`min_cps_rate`,`max_cps_rate`)
  - `target_setup_duration` = `target_session_number` / `trial_cps_rate`
  - For UDP:
    - `trial_duration` = `target_setup_duration` + `late_traffic_start_correction`
    - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now)
  - For TCP:
    - `trial_duration` = 2 * `target_setup_duration` + `late_traffic_start_correction`
    - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now)

## UDP CPS Tests

### UDP TRex Measurements

#### Counters

Following TRex ASTF counters are collected by UDP CPS tests for automated
results evaluation (r) and debugging purposes (d):

- Interface 1 Client
  - (r) `opackets`, TRex UDP transaction start
  - (r) `ipackets`, TRex UDP transaction finish
- Interface 2 Server
  - (d) `opackets`
  - (d) `ipackets`
- Traffic Client
  - (d) `m_active_flows`
  - (d) `m_est_flows`
  - (d) `m_traffic_duration`, includes TRex ramp-up overhead, and it can
    be quite far from the actual traffic duration
  - (d) `udps_connects`
  - (d) `udps_closed`
  - (d) `udps_sndbyte`
  - (d) `udps_sndpkt`
  - (d) `udps_rcvbyte`
  - (d) `udps_rcvpkt`
  - (d) `udps_keepdrops`, TRex out of capacity, dropping UDP KAs(?)
<!--
Vratko Polak: Yes, although the traffic profile should have set large
enough keepalive value so zero KA packets are actually sent within the
trial. I did not actually check the value is large enough for the worst
case (ndrpdr search hitting min multiplier of 9001).
-->
  - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due
    to Rx overload(?)
<!--
Vratko Polak: I think this is TRex receiving the packet on L2 level, but
then dropping it because L7 buffers are full. Such packet increases
ipackets, but does not increase any L7 counter (even if traffic profile
wants to receive that packet). But this is just me guessing. TRex docs
say "rx thread was throttled due too many packets in NIC rx queue", and
I did no experiments/investigation to confirm my hypothesis fits with
the observed counters.
-->
  - (d) `err_c_nf_throttled`, Number of client side flows that were not
    opened due to flow-table overflow(?)
  - (d) `err_flow_overflow`, too many flows(?)
- Traffic Server
  - (d) `m_active_flows`
  - (d) `m_est_flows`
  - (r) `m_traffic_duration`
  - (d) `udps_accepts`
  - (d) `udps_closed`
  - (d) `udps_sndbyte`
  - (d) `udps_sndpkt`
  - (d) `udps_rcvbyte`
  - (d) `udps_rcvpkt`
  - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due
    to Rx overload(?)

[TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference).

TRex counters are polled once TRex confirms traffic is stopped, after it
is explicitly instructed to stop it. Early attempts to use periodic TRex
counter polling affected TRex behaviour and test results, hence counter
polling is consider as invasive.

#### Calculations

- Interface packet loss
  - pktloss_ratio = (c_opackets - c_ipackets) / c_opackets
- UDP session packet loss (currently not used)
- UDP session byte loss (currently not used)
- UDP session integrity (currently not used)

#### CPS-MRR

Reported MRR values are calculated as follows:

CPS-MRR = `c_ipackets` / `s_traffic_duration`, where
`s_traffic_duration` = TRex Traffic Server `m_traffic_duration`.

In order to ensure a determnistic region of TRex ASTF operation, a
separate set of tests is run for each traffic profile, with vpp-ip4base
DUT instead of vpp-nat44ed, to auto-discover the maximum rate TRex ASTF
traffic profile is capable of. Result of this test is used as a side
reference to compare with the results of NAT44ed CPS-MRR tests.

#### CPS-PDR

CPS-PDR values are discovered using MLRsearch, a binary search optimized
for the overall test duration.

CPS-PDR = max(`trial_cps_rate`) found for `pktloss_ratio` <
`target_loss_ratio`, according to MLRsearch criteria for PDR.

Measurements to be reported in the CPS-PDR result test message:

- PDR_LOWER

#### CPS-NDR

CPS-NDR values are also discovered using MLRsearch.

CPS-NDR = max(`trial_cps_rate`) found for `pktloss_ratio` = 0, according
to MLRsearch criteria for PDR.

Measurements to be reported in the CPS-NDR result test message:

- NDR_LOWER

### UDP VPP Telemetry

#### Counters

- VPP show nat44 summary

  ```
  max translations per thread: 81920
  max translations per user: 81920
  total timed out sessions: 0
  total sessions: 64514
  total tcp sessions: 0
  total tcp established sessions: 0
  total tcp transitory sessions: 0
  total tcp transitory (WAIT-CLOSED) sessions: 0
  total tcp transitory (CLOSED) sessions: 0
  total udp sessions: 64514
  total icmp sessions: 0
  ```

- VPP show interface

  ```
  show hardware verbose (10.30.51.54 - /run/vpp/api.sock):
  Name                Idx   Link  Hardware
  avf-0/3b/2/0                       1     up   avf-0/3b/2/0
    Link speed: 25 Gbps
    Ethernet address 3c:fe:bd:f9:00:00
    flags: initialized admin-up vaddr-dma link-up rx-interrupts
    offload features: l2 vlan rx-polling rss-pf
    num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
    speed
    stats:
      rx bytes             69368896
      rx unicast           135301620
      rx discards          94585780
      tx bytes             2401281120
      tx unicast           40021352
  avf-0/3b/a/0                       2     up   avf-0/3b/a/0
    Link speed: 25 Gbps
    Ethernet address 3c:fe:bd:f9:01:00
    flags: initialized admin-up vaddr-dma link-up rx-interrupts
    offload features: l2 vlan rx-polling rss-pf
    num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64
    speed
    stats:
      rx bytes             40912192
      rx unicast           134856987
      rx discards          94835635
      tx bytes             2442955680
      tx unicast           40715928
  ```

- VPP show runtime

  ```
  Thread 1 vpp_wk_0 (lcore 2)
  Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6740197.88
    vector rates in 4.2183e3, out 3.7118e3, drop 0.0000e0, punt 0.0000e0
               Name                 State         Calls          Vectors        Suspends         Clocks       Vectors/Call
  avf-0/3b/2/0-output              active                277           34387               0          1.96e1          124.14
  avf-0/3b/2/0-tx                  active                277           34387               0          3.54e1          124.14
  avf-0/3b/a/0-output              active                380           45245               0          1.92e1          119.07
  avf-0/3b/a/0-tx                  active                380           45245               0          3.36e1          119.07
  avf-input                        polling         144384995           90499               0          3.03e5            0.00
  ethernet-input                   active                381           90499               0          1.91e1          237.53
  ip4-input-no-checksum            active                381           90499               0          4.94e1          237.53
  ip4-lookup                       active                521           79632               0          3.76e1          152.84
  ip4-rewrite                      active                521           79632               0          4.19e1          152.84
  ip4-sv-reassembly-feature        active                381           90499               0          3.78e1          237.53
  nat44-ed-in2out                  active                380           45245               0          1.98e2          119.07
  nat44-ed-in2out-slowpath         active                380           45245               0          2.31e3          119.07
  nat44-ed-out2in                  active                277           34387               0          1.89e2          124.14
  nat44-in2out-worker-handoff      active                381           90499               0          9.42e1          237.53
  unix-epoll-input                 polling            140863               0               0          1.61e3            0.00
  ---------------
  Thread 2 vpp_wk_1 (lcore 58)
  Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6733488.17
    vector rates in 3.3365e3, out 3.5604e3, drop 0.0000e0, punt 0.0000e0
               Name                 State         Calls          Vectors        Suspends         Clocks       Vectors/Call
  avf-0/3b/2/0-output              active                276           31129               0          2.03e1          112.79
  avf-0/3b/2/0-tx                  active                276           31129               0          3.63e1          112.79
  avf-0/3b/a/0-output              active                332           45254               0          1.87e1          136.31
  avf-0/3b/a/0-tx                  active                332           45254               0          3.48e1          136.31
  avf-input                        polling         166439403           71581               0          4.42e5            0.00
  ethernet-input                   active                277           65516               0          1.89e1          236.52
  ip4-input-no-checksum            active                277           65516               0          4.95e1          236.52
  ip4-lookup                       active                455           76383               0          3.75e1          167.87
  ip4-rewrite                      active                455           76383               0          4.20e1          167.87
  ip4-sv-reassembly-feature        active                277           65516               0          3.85e1          236.52
  nat44-ed-in2out                  active                377           45254               0          1.97e2          120.04
  nat44-ed-in2out-slowpath         active                332           45254               0          2.39e3          136.31
  nat44-ed-out2in                  active                276           31129               0          1.83e2          112.79
  nat44-out2in-worker-handoff      active                277           65516               0          2.17e2          236.52
  unix-epoll-input                 polling            140817               0               0          1.60e3            0.00
  ```

#### Errors

- VPP show errors

  ```
  Count                    Node                  Reason
       32258       nat44-in2out-worker-handoff      same worker
       32256       nat44-in2out-worker-handoff      do handoff
       32258             nat44-ed-out2in            good out2in packets processed
       32258             nat44-ed-out2in            UDP packets
       32258        nat44-ed-in2out-slowpath        good in2out packets processed
       32258        nat44-ed-in2out-slowpath        UDP packets
       32256       nat44-out2in-worker-handoff      same worker
       32258       nat44-out2in-worker-handoff      do handoff
       32256             nat44-ed-out2in            good out2in packets processed
       32256             nat44-ed-out2in            UDP packets
       32256        nat44-ed-in2out-slowpath        good in2out packets processed
       32256        nat44-ed-in2out-slowpath        UDP packets
  ```

## TCP/IP CPS Tests

### TCP/IP TRex Measurements

#### Counters

Following TRex ASTF counters are collected by UDP CPS tests for automated
results evaluation (r) and debugging purposes (d):

- Interface 1 Client
  - (d) `opackets`
  - (d) `packets`
- Interface 2 Server
  - (d) `opackets`
  - (d) `packets`
- Traffic Client
  - (d) `m_active_flows`
  - (d) `m_est_flows`
  - (d) `m_traffic_duration`
  - (r) `tcps_connattempt`
  - (d) `tcps_connects`
  - (d) `tcps_closed`
- Traffic Server
  - (d) `m_active_flows`
  - (d) `m_est_flows`
  - (r) `m_traffic_duration`
  - (d) `tcps_accepts`
  - (r) `tcps_connects`
  - (d) `tcps_closed`
  - (d) `err_no_template`, server can’t match L7 template no destination port or IP range

[TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference).

TRex counters are polled only once by CSIT after traffic is stopped.

#### Calculations

TODO WIP Note: Currently s_tcp_connects is used for counting successful
sessions. But now I am not sure whether it is correct, as already
c_tcps_connects counts NAT sessions that got established (even though
TCP is not fully connected yet). Not sure how the counters behave when
the third packet is lost and retransmitted.

- Interface packet loss
  - `pktloss_c_s` = `c_opackets` - `s_ipackets`
  - `pktloss_s_c` = `s_opackets` - `c_ipackets`
  - `pktloss_ratio` = (`pktloss_s_c` + `pktloss_c_s`) / (`c_opackets` + `s_opackets`)
- TCP session integrity
  - `tcp_attempted_connection_count` = `c_tcps_connattempt`
  - `tcp_failed_connection_count` = `c_tcps_connects` - `c_tcps_connattempt`

#### CPS Trial PASS

TODO WIP Note: Currently any trial measurement fails only if TRex itself
fails, or if we fail to parse some counter. No criteria mentioned here
is currently planned to be implemented; we rely on bad things leading to
too few (maybe zero) passed transactions.

<!--
PASS of TCP CPS test trial is conditioned on all of the following criteria being met:

- PASS-C1 TRex must attempt all configured `target_session_number` in `target_setup_duration` time
   - IOW TRex must send connect packets at configured `trial_cps_rate`.
- PASS-C2 Following TRex errors ARE NOT recorded in Target-Counters:
  - Traffic Client
    - No errors recorded so far
  - Traffic Server
    - `err_no_template`, server can’t match L7 template no destination port or IP range
-->

#### CPS-MRR

Reported MRR values are equal to the following TRex counters from Target-Counters:
- `c_m_est_flows`
- `s_m_est_flows`

TODO Add description of separate set of tests for discovering a **safe**
CPS-MTR value (Maximum Transmit Rate) for TRex, where TRex errors **are not**
observed in Target-Counters.

#### CPS-PDR

CPS-PDR values are discovered using MLRsearch, a binary search optimized
for the overall test duration.

CPS-PDR = `trial_cps_rate`, if all of the following conditions are met:

- `tcp_failed_connection_count` < `target_loss_ratio`
- `pktloss_ratio` < `target_loss_ratio`

Measurements to be reported in the CPS-PDR result test message:

- `trial_cps_rate`
- `c_m_est_flows`
- `s_m_est_flows`

#### CPS-NDR

CPS-NDR values are discovered using MLRsearch, a binary search optimized
for the overall test duration.

CPS-NDR = `trial_cps_rate`, if all of the following conditions are met:

- `tcp_failed_connection_count` = 0
- `pktloss_ratio` = 0

Measurements to be reported in the CPS-PDR result test message:

- `trial_cps_rate`
- `c_m_est_flows`
- `s_m_est_flows`

### TCP/IP VPP Telemetry

#### Counters

- VPP show nat44 summary

  ```
  <TODO add sample output>
  ```

- VPP show interface

  ```
  <TODO add sample output>
  ```

- VPP show runtime

  ```
  <TODO add sample output>
  ```

#### Errors

- VPP show errors

  ```
  <TODO add sample output>
  ```