## Content - [Tests for NAT44ED](#tests-for-nat44ed) - [CPS Test Objectives](#cps-test-objectives) - [Input Parameters](#input-parameters) - [Stateful traffic profiles](#stateful-traffic-profiles) - [UDP CPS Tests](#udp-cps-tests) - [UDP TRex Measurements](#udp-trex-measurements) - [Counters](#counters) - [Calculations](#calculations) - [CPS-MRR](#cps-mrr) - [CPS-PDR](#cps-pdr) - [CPS-NDR](#cps-ndr) - [UDP VPP Telemetry](#udp-vpp-telemetry) - [Counters](#counters-1) - [Errors](#errors) - [TCP/IP CPS Tests](#tcpip-cps-tests) - [TCP/IP TRex Measurements](#tcpip-trex-measurements) - [Counters](#counters-2) - [Calculations](#calculations-1) - [CPS Trial PASS](#cps-trial-pass) - [CPS-MRR](#cps-mrr-1) - [CPS-PDR](#cps-pdr-1) - [CPS-NDR](#cps-ndr-1) - [TCP/IP VPP Telemetry](#tcpip-vpp-telemetry) - [Counters](#counters-3) - [Errors](#errors-1) ## Tests for NAT44ED Two types of stateful tests are developed for NAT44ED (source network address and port translation IPv4 to IPv4 with 5-tuple session state): - Connections-Per-Second (CPS), discovering the maximum rate of creating NAT44ED sessions. Measured separately for UDP and TCP connections and for different session scale. - Packets-Per-Second (PPS), discovering the maximum rate of simultaneously creating NAT44ED sessions and transfering bulk of data packets across the corresponding connections. Measured separately for UDP and TCP connections with different session scale and different data packet sizes per each connection. Current code is using 64B only for UDP and default MSS 1460B for TCP/IP. This note describes CPS tests. ## CPS Test Objectives Discover DUT's highest sustain rate of creating fully functional NAT44ED 5-tuple stateful session entries. Session entry is considered fully functional, if packets associated with this entry are NAT44ED processed by DUT and forwarded in both directions without loss. Similarly to packet throughput tests, three CPS rates are discovered: - CPS-MRR, verified connection rate at maximal connection attempt rate, regardless of an amount of not established connections. (Connections per Second - Maximum Receive Rate.) - CPS-NDR, maximal connection attempt rate at which all connections get established. (Connections per Second - Non Drop Rate.) - CPS-PDR, maximal connection attempt rate at which ratio of not established connections to attempted connections is below configured threshold. (Connections per Second - Partial Drop Rate.) ## Input Parameters - `max_cps_rate`, maximum rate of attempting connections, to be used by traffic generator, limited by traffic generator capabilities, Ethernet link(s) rate and NIC model. - `min_cps_rate`, minimum rate of establishing connections to be used for measurements. Search fails if lower transmit rate needs to be used to meet search criteria. - `target_session_number`, maximum number of sessions to be established and tested. - `target_loss_ratio`, maximum acceptable connections loss ratio search criteria for PDR measurements with UDP tests. Indicates packet drop impact on connection establishment rate. - `final_relative_width`, required measurement resolution expressed as (lower_bound, upper_bound) interval width relative to upper_bound. - stateful traffic profiles, TRex ASTF program defining the connection per L4 protocol tested (TCP, UDP), including connect and close sequence. ## Stateful traffic profiles TRex ASTF program defines following TCP and UDP transactions for discovering NAT44ED CPS limits: - CPS with TCP - connect(syn,syn-ack,ack) - pkts client tx 2, rx 1 - pkts server tx 1, rx 2 - delay (note: optional, currently not implemented) - no packets - close(fin,fin-ack,ack,ack) - pkts client tx 2, rx 2 - pkts server tx 1, rx 2 - CPS with UDP - connect_and_close(req,ack) - pkts client tx 1, rx 1 - pkts server tx 1, rx 1 TRex ASTF program configuration parameters: - `limit` of connections, set to `target_session_number`. - `multiplier`, represents `trial_cps_rate`, a number of connections per second to be executed per trial. Multiplier applies to connect phases. Close phases occur automatically based on arrival of the last packet expected per session. - IPv4 source and destination address and port ranges matching the limit of connections. - Source and destination addresses changing packet-by-packet with two separate profiles i) incrementing sequentially pair-wise (implemented) and ii) changed randomly (with seed) pair-wise (not implemented yet). - Source port changing randomly within the range. - `trial_duration`, function of `target_session_number` and `multiplier` - `multiplier`, subject of the search, value in the range (`min_cps_rate`,`max_cps_rate`) - `target_setup_duration` = `target_session_number` / `trial_cps_rate` - For UDP: - `trial_duration` = `target_setup_duration` + `late_traffic_start_correction` - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now) - For TCP: - `trial_duration` = 2 * `target_setup_duration` + `late_traffic_start_correction` - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now) ## UDP CPS Tests ### UDP TRex Measurements #### Counters Following TRex ASTF counters are collected by UDP CPS tests for automated results evaluation (r) and debugging purposes (d): - Interface 1 Client - (r) `opackets`, TRex UDP transaction start - (r) `ipackets`, TRex UDP transaction finish - Interface 2 Server - (d) `opackets` - (d) `ipackets` - Traffic Client - (d) `m_active_flows` - (d) `m_est_flows` - (d) `m_traffic_duration`, includes TRex ramp-up overhead, and it can be quite far from the actual traffic duration - (d) `udps_connects` - (d) `udps_closed` - (d) `udps_sndbyte` - (d) `udps_sndpkt` - (d) `udps_rcvbyte` - (d) `udps_rcvpkt` - (d) `udps_keepdrops`, TRex out of capacity, dropping UDP KAs(?) - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due to Rx overload(?) - (d) `err_c_nf_throttled`, Number of client side flows that were not opened due to flow-table overflow(?) - (d) `err_flow_overflow`, too many flows(?) - Traffic Server - (d) `m_active_flows` - (d) `m_est_flows` - (r) `m_traffic_duration` - (d) `udps_accepts` - (d) `udps_closed` - (d) `udps_sndbyte` - (d) `udps_sndpkt` - (d) `udps_rcvbyte` - (d) `udps_rcvpkt` - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due to Rx overload(?) [TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference). TRex counters are polled once TRex confirms traffic is stopped, after it is explicitly instructed to stop it. Early attempts to use periodic TRex counter polling affected TRex behaviour and test results, hence counter polling is consider as invasive. #### Calculations - Interface packet loss - pktloss_ratio = (c_opackets - c_ipackets) / c_opackets - UDP session packet loss (currently not used) - UDP session byte loss (currently not used) - UDP session integrity (currently not used) #### CPS-MRR Reported MRR values are calculated as follows: CPS-MRR = `c_ipackets` / `s_traffic_duration`, where `s_traffic_duration` = TRex Traffic Server `m_traffic_duration`. In order to ensure a determnistic region of TRex ASTF operation, a separate set of tests is run for each traffic profile, with vpp-ip4base DUT instead of vpp-nat44ed, to auto-discover the maximum rate TRex ASTF traffic profile is capable of. Result of this test is used as a side reference to compare with the results of NAT44ed CPS-MRR tests. #### CPS-PDR CPS-PDR values are discovered using MLRsearch, a binary search optimized for the overall test duration. CPS-PDR = max(`trial_cps_rate`) found for `pktloss_ratio` < `target_loss_ratio`, according to MLRsearch criteria for PDR. Measurements to be reported in the CPS-PDR result test message: - PDR_LOWER #### CPS-NDR CPS-NDR values are also discovered using MLRsearch. CPS-NDR = max(`trial_cps_rate`) found for `pktloss_ratio` = 0, according to MLRsearch criteria for PDR. Measurements to be reported in the CPS-NDR result test message: - NDR_LOWER ### UDP VPP Telemetry #### Counters - VPP show nat44 summary ``` max translations per thread: 81920 max translations per user: 81920 total timed out sessions: 0 total sessions: 64514 total tcp sessions: 0 total tcp established sessions: 0 total tcp transitory sessions: 0 total tcp transitory (WAIT-CLOSED) sessions: 0 total tcp transitory (CLOSED) sessions: 0 total udp sessions: 64514 total icmp sessions: 0 ``` - VPP show interface ``` show hardware verbose (10.30.51.54 - /run/vpp/api.sock): Name Idx Link Hardware avf-0/3b/2/0 1 up avf-0/3b/2/0 Link speed: 25 Gbps Ethernet address 3c:fe:bd:f9:00:00 flags: initialized admin-up vaddr-dma link-up rx-interrupts offload features: l2 vlan rx-polling rss-pf num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64 speed stats: rx bytes 69368896 rx unicast 135301620 rx discards 94585780 tx bytes 2401281120 tx unicast 40021352 avf-0/3b/a/0 2 up avf-0/3b/a/0 Link speed: 25 Gbps Ethernet address 3c:fe:bd:f9:01:00 flags: initialized admin-up vaddr-dma link-up rx-interrupts offload features: l2 vlan rx-polling rss-pf num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64 speed stats: rx bytes 40912192 rx unicast 134856987 rx discards 94835635 tx bytes 2442955680 tx unicast 40715928 ``` - VPP show runtime ``` Thread 1 vpp_wk_0 (lcore 2) Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6740197.88 vector rates in 4.2183e3, out 3.7118e3, drop 0.0000e0, punt 0.0000e0 Name State Calls Vectors Suspends Clocks Vectors/Call avf-0/3b/2/0-output active 277 34387 0 1.96e1 124.14 avf-0/3b/2/0-tx active 277 34387 0 3.54e1 124.14 avf-0/3b/a/0-output active 380 45245 0 1.92e1 119.07 avf-0/3b/a/0-tx active 380 45245 0 3.36e1 119.07 avf-input polling 144384995 90499 0 3.03e5 0.00 ethernet-input active 381 90499 0 1.91e1 237.53 ip4-input-no-checksum active 381 90499 0 4.94e1 237.53 ip4-lookup active 521 79632 0 3.76e1 152.84 ip4-rewrite active 521 79632 0 4.19e1 152.84 ip4-sv-reassembly-feature active 381 90499 0 3.78e1 237.53 nat44-ed-in2out active 380 45245 0 1.98e2 119.07 nat44-ed-in2out-slowpath active 380 45245 0 2.31e3 119.07 nat44-ed-out2in active 277 34387 0 1.89e2 124.14 nat44-in2out-worker-handoff active 381 90499 0 9.42e1 237.53 unix-epoll-input polling 140863 0 0 1.61e3 0.00 --------------- Thread 2 vpp_wk_1 (lcore 58) Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6733488.17 vector rates in 3.3365e3, out 3.5604e3, drop 0.0000e0, punt 0.0000e0 Name State Calls Vectors Suspends Clocks Vectors/Call avf-0/3b/2/0-output active 276 31129 0 2.03e1 112.79 avf-0/3b/2/0-tx active 276 31129 0 3.63e1 112.79 avf-0/3b/a/0-output active 332 45254 0 1.87e1 136.31 avf-0/3b/a/0-tx active 332 45254 0 3.48e1 136.31 avf-input polling 166439403 71581 0 4.42e5 0.00 ethernet-input active 277 65516 0 1.89e1 236.52 ip4-input-no-checksum active 277 65516 0 4.95e1 236.52 ip4-lookup active 455 76383 0 3.75e1 167.87 ip4-rewrite active 455 76383 0 4.20e1 167.87 ip4-sv-reassembly-feature active 277 65516 0 3.85e1 236.52 nat44-ed-in2out active 377 45254 0 1.97e2 120.04 nat44-ed-in2out-slowpath active 332 45254 0 2.39e3 136.31 nat44-ed-out2in active 276 31129 0 1.83e2 112.79 nat44-out2in-worker-handoff active 277 65516 0 2.17e2 236.52 unix-epoll-input polling 140817 0 0 1.60e3 0.00 ``` #### Errors - VPP show errors ``` Count Node Reason 32258 nat44-in2out-worker-handoff same worker 32256 nat44-in2out-worker-handoff do handoff 32258 nat44-ed-out2in good out2in packets processed 32258 nat44-ed-out2in UDP packets 32258 nat44-ed-in2out-slowpath good in2out packets processed 32258 nat44-ed-in2out-slowpath UDP packets 32256 nat44-out2in-worker-handoff same worker 32258 nat44-out2in-worker-handoff do handoff 32256 nat44-ed-out2in good out2in packets processed 32256 nat44-ed-out2in UDP packets 32256 nat44-ed-in2out-slowpath good in2out packets processed 32256 nat44-ed-in2out-slowpath UDP packets ``` ## TCP/IP CPS Tests ### TCP/IP TRex Measurements #### Counters Following TRex ASTF counters are collected by UDP CPS tests for automated results evaluation (r) and debugging purposes (d): - Interface 1 Client - (d) `opackets` - (d) `packets` - Interface 2 Server - (d) `opackets` - (d) `packets` - Traffic Client - (d) `m_active_flows` - (d) `m_est_flows` - (d) `m_traffic_duration` - (r) `tcps_connattempt` - (d) `tcps_connects` - (d) `tcps_closed` - Traffic Server - (d) `m_active_flows` - (d) `m_est_flows` - (r) `m_traffic_duration` - (d) `tcps_accepts` - (r) `tcps_connects` - (d) `tcps_closed` - (d) `err_no_template`, server can’t match L7 template no destination port or IP range [TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference). TRex counters are polled only once by CSIT after traffic is stopped. #### Calculations TODO WIP Note: Currently s_tcp_connects is used for counting successful sessions. But now I am not sure whether it is correct, as already c_tcps_connects counts NAT sessions that got established (even though TCP is not fully connected yet). Not sure how the counters behave when the third packet is lost and retransmitted. - Interface packet loss - `pktloss_c_s` = `c_opackets` - `s_ipackets` - `pktloss_s_c` = `s_opackets` - `c_ipackets` - `pktloss_ratio` = (`pktloss_s_c` + `pktloss_c_s`) / (`c_opackets` + `s_opackets`) - TCP session integrity - `tcp_attempted_connection_count` = `c_tcps_connattempt` - `tcp_failed_connection_count` = `c_tcps_connects` - `c_tcps_connattempt` #### CPS Trial PASS TODO WIP Note: Currently any trial measurement fails only if TRex itself fails, or if we fail to parse some counter. No criteria mentioned here is currently planned to be implemented; we rely on bad things leading to too few (maybe zero) passed transactions. #### CPS-MRR Reported MRR values are equal to the following TRex counters from Target-Counters: - `c_m_est_flows` - `s_m_est_flows` TODO Add description of separate set of tests for discovering a **safe** CPS-MTR value (Maximum Transmit Rate) for TRex, where TRex errors **are not** observed in Target-Counters. #### CPS-PDR CPS-PDR values are discovered using MLRsearch, a binary search optimized for the overall test duration. CPS-PDR = `trial_cps_rate`, if all of the following conditions are met: - `tcp_failed_connection_count` < `target_loss_ratio` - `pktloss_ratio` < `target_loss_ratio` Measurements to be reported in the CPS-PDR result test message: - `trial_cps_rate` - `c_m_est_flows` - `s_m_est_flows` #### CPS-NDR CPS-NDR values are discovered using MLRsearch, a binary search optimized for the overall test duration. CPS-NDR = `trial_cps_rate`, if all of the following conditions are met: - `tcp_failed_connection_count` = 0 - `pktloss_ratio` = 0 Measurements to be reported in the CPS-PDR result test message: - `trial_cps_rate` - `c_m_est_flows` - `s_m_est_flows` ### TCP/IP VPP Telemetry #### Counters - VPP show nat44 summary ``` ``` - VPP show interface ``` ``` - VPP show runtime ``` ``` #### Errors - VPP show errors ``` ```