diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/test_specs/csit-nat44ed-cps-spec.md | 499 |
1 files changed, 499 insertions, 0 deletions
diff --git a/docs/test_specs/csit-nat44ed-cps-spec.md b/docs/test_specs/csit-nat44ed-cps-spec.md new file mode 100644 index 0000000000..c39ba9eced --- /dev/null +++ b/docs/test_specs/csit-nat44ed-cps-spec.md @@ -0,0 +1,499 @@ +## Content + +<!-- MarkdownTOC autolink="true" --> + +- [Tests for NAT44ED](#tests-for-nat44ed) +- [CPS Test Objectives](#cps-test-objectives) +- [Input Parameters](#input-parameters) +- [Stateful traffic profiles](#stateful-traffic-profiles) +- [UDP CPS Tests](#udp-cps-tests) + - [UDP TRex Measurements](#udp-trex-measurements) + - [Counters](#counters) + - [Calculations](#calculations) + - [CPS-MRR](#cps-mrr) + - [CPS-PDR](#cps-pdr) + - [CPS-NDR](#cps-ndr) + - [UDP VPP Telemetry](#udp-vpp-telemetry) + - [Counters](#counters-1) + - [Errors](#errors) +- [TCP/IP CPS Tests](#tcpip-cps-tests) + - [TCP/IP TRex Measurements](#tcpip-trex-measurements) + - [Counters](#counters-2) + - [Calculations](#calculations-1) + - [CPS Trial PASS](#cps-trial-pass) + - [CPS-MRR](#cps-mrr-1) + - [CPS-PDR](#cps-pdr-1) + - [CPS-NDR](#cps-ndr-1) + - [TCP/IP VPP Telemetry](#tcpip-vpp-telemetry) + - [Counters](#counters-3) + - [Errors](#errors-1) + +<!-- /MarkdownTOC --> + +## Tests for NAT44ED + +Two types of stateful tests are developed for NAT44ED (source network address +and port translation IPv4 to IPv4 with 5-tuple session state): + +- Connections-Per-Second (CPS), discovering the maximum rate of creating + NAT44ED sessions. Measured separately for UDP and TCP connections and + for different session scale. + +- Packets-Per-Second (PPS), discovering the maximum rate of +simultaneously creating NAT44ED sessions and transfering bulk of data +packets across the corresponding connections. Measured separately for +UDP and TCP connections with different session scale and different data +packet sizes per each connection. Current code is using 64B only for UDP +and default MSS 1460B for TCP/IP. + +This note describes CPS tests. + +## CPS Test Objectives + +Discover DUT's highest sustain rate of creating fully functional NAT44ED +5-tuple stateful session entries. Session entry is considered fully +functional, if packets associated with this entry are NAT44ED processed +by DUT and forwarded in both directions without loss. + +Similarly to packet throughput tests, three CPS rates are discovered: + +- CPS-MRR, verified connection rate at maximal connection attempt rate, + regardless of an amount of not established connections. (Connections + per Second - Maximum Receive Rate.) +- CPS-NDR, maximal connection attempt rate at which all connections get + established. (Connections per Second - Non Drop Rate.) +- CPS-PDR, maximal connection attempt rate at which ratio of not + established connections to attempted connections is below configured + threshold. (Connections per Second - Partial Drop Rate.) + +## Input Parameters + +- `max_cps_rate`, maximum rate of attempting connections, to be used by + traffic generator, limited by traffic generator capabilities, Ethernet + link(s) rate and NIC model. +- `min_cps_rate`, minimum rate of establishing connections to be used for + measurements. Search fails if lower transmit rate needs to be used to + meet search criteria. +- `target_session_number`, maximum number of sessions to be established and + tested. +- `target_loss_ratio`, maximum acceptable connections loss ratio search + criteria for PDR measurements with UDP tests. Indicates packet drop + impact on connection establishment rate. +- `final_relative_width`, required measurement resolution expressed as + (lower_bound, upper_bound) interval width relative to upper_bound. +- stateful traffic profiles, TRex ASTF program defining the connection + per L4 protocol tested (TCP, UDP), including connect and + close sequence. + +## Stateful traffic profiles + +TRex ASTF program defines following TCP and UDP transactions for +discovering NAT44ED CPS limits: + +- CPS with TCP + - connect(syn,syn-ack,ack) + - pkts client tx 2, rx 1 + - pkts server tx 1, rx 2 + - delay (note: optional, currently not implemented) + - no packets + - close(fin,fin-ack,ack,ack) + - pkts client tx 2, rx 2 + - pkts server tx 1, rx 2 +- CPS with UDP + - connect_and_close(req,ack) + - pkts client tx 1, rx 1 + - pkts server tx 1, rx 1 + +TRex ASTF program configuration parameters: + +- `limit` of connections, set to `target_session_number`. +- `multiplier`, represents `trial_cps_rate`, a number of connections per + second to be executed per trial. Multiplier applies to connect phases. + Close phases occur automatically based on arrival of the last packet + expected per session. +- IPv4 source and destination address and port ranges matching the + limit of connections. + - Source and destination addresses changing packet-by-packet with two + separate profiles i) incrementing sequentially pair-wise + (implemented) and ii) changed randomly (with seed) pair-wise (not + implemented yet). + - Source port changing randomly within the range. +- `trial_duration`, function of `target_session_number` and `multiplier` + - `multiplier`, subject of the search, value in the range (`min_cps_rate`,`max_cps_rate`) + - `target_setup_duration` = `target_session_number` / `trial_cps_rate` + - For UDP: + - `trial_duration` = `target_setup_duration` + `late_traffic_start_correction` + - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now) + - For TCP: + - `trial_duration` = 2 * `target_setup_duration` + `late_traffic_start_correction` + - `late_traffic_start_correction` = 0.1115 seconds (hardcoded for now) + +## UDP CPS Tests + +### UDP TRex Measurements + +#### Counters + +Following TRex ASTF counters are collected by UDP CPS tests for automated +results evaluation (r) and debugging purposes (d): + +- Interface 1 Client + - (r) `opackets`, TRex UDP transaction start + - (r) `ipackets`, TRex UDP transaction finish +- Interface 2 Server + - (d) `opackets` + - (d) `ipackets` +- Traffic Client + - (d) `m_active_flows` + - (d) `m_est_flows` + - (d) `m_traffic_duration`, includes TRex ramp-up overhead, and it can + be quite far from the actual traffic duration + - (d) `udps_connects` + - (d) `udps_closed` + - (d) `udps_sndbyte` + - (d) `udps_sndpkt` + - (d) `udps_rcvbyte` + - (d) `udps_rcvpkt` + - (d) `udps_keepdrops`, TRex out of capacity, dropping UDP KAs(?) +<!-- +Vratko Polak: Yes, although the traffic profile should have set large +enough keepalive value so zero KA packets are actually sent within the +trial. I did not actually check the value is large enough for the worst +case (ndrpdr search hitting min multiplier of 9001). +--> + - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due + to Rx overload(?) +<!-- +Vratko Polak: I think this is TRex receiving the packet on L2 level, but +then dropping it because L7 buffers are full. Such packet increases +ipackets, but does not increase any L7 counter (even if traffic profile +wants to receive that packet). But this is just me guessing. TRex docs +say "rx thread was throttled due too many packets in NIC rx queue", and +I did no experiments/investigation to confirm my hypothesis fits with +the observed counters. +--> + - (d) `err_c_nf_throttled`, Number of client side flows that were not + opened due to flow-table overflow(?) + - (d) `err_flow_overflow`, too many flows(?) +- Traffic Server + - (d) `m_active_flows` + - (d) `m_est_flows` + - (r) `m_traffic_duration` + - (d) `udps_accepts` + - (d) `udps_closed` + - (d) `udps_sndbyte` + - (d) `udps_sndpkt` + - (d) `udps_rcvbyte` + - (d) `udps_rcvpkt` + - (d) `err_rx_throttled`, TRex out of capacity, throttling workers due + to Rx overload(?) + +[TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference). + +TRex counters are polled once TRex confirms traffic is stopped, after it +is explicitly instructed to stop it. Early attempts to use periodic TRex +counter polling affected TRex behaviour and test results, hence counter +polling is consider as invasive. + +#### Calculations + +- Interface packet loss + - pktloss_ratio = (c_opackets - c_ipackets) / c_opackets +- UDP session packet loss (currently not used) +- UDP session byte loss (currently not used) +- UDP session integrity (currently not used) + +#### CPS-MRR + +Reported MRR values are calculated as follows: + +CPS-MRR = `c_ipackets` / `s_traffic_duration`, where +`s_traffic_duration` = TRex Traffic Server `m_traffic_duration`. + +In order to ensure a determnistic region of TRex ASTF operation, a +separate set of tests is run for each traffic profile, with vpp-ip4base +DUT instead of vpp-nat44ed, to auto-discover the maximum rate TRex ASTF +traffic profile is capable of. Result of this test is used as a side +reference to compare with the results of NAT44ed CPS-MRR tests. + +#### CPS-PDR + +CPS-PDR values are discovered using MLRsearch, a binary search optimized +for the overall test duration. + +CPS-PDR = max(`trial_cps_rate`) found for `pktloss_ratio` < +`target_loss_ratio`, according to MLRsearch criteria for PDR. + +Measurements to be reported in the CPS-PDR result test message: + +- PDR_LOWER + +#### CPS-NDR + +CPS-NDR values are also discovered using MLRsearch. + +CPS-NDR = max(`trial_cps_rate`) found for `pktloss_ratio` = 0, according +to MLRsearch criteria for PDR. + +Measurements to be reported in the CPS-NDR result test message: + +- NDR_LOWER + +### UDP VPP Telemetry + +#### Counters + +- VPP show nat44 summary + + ``` + max translations per thread: 81920 + max translations per user: 81920 + total timed out sessions: 0 + total sessions: 64514 + total tcp sessions: 0 + total tcp established sessions: 0 + total tcp transitory sessions: 0 + total tcp transitory (WAIT-CLOSED) sessions: 0 + total tcp transitory (CLOSED) sessions: 0 + total udp sessions: 64514 + total icmp sessions: 0 + ``` + +- VPP show interface + + ``` + show hardware verbose (10.30.51.54 - /run/vpp/api.sock): + Name Idx Link Hardware + avf-0/3b/2/0 1 up avf-0/3b/2/0 + Link speed: 25 Gbps + Ethernet address 3c:fe:bd:f9:00:00 + flags: initialized admin-up vaddr-dma link-up rx-interrupts + offload features: l2 vlan rx-polling rss-pf + num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64 + speed + stats: + rx bytes 69368896 + rx unicast 135301620 + rx discards 94585780 + tx bytes 2401281120 + tx unicast 40021352 + avf-0/3b/a/0 2 up avf-0/3b/a/0 + Link speed: 25 Gbps + Ethernet address 3c:fe:bd:f9:01:00 + flags: initialized admin-up vaddr-dma link-up rx-interrupts + offload features: l2 vlan rx-polling rss-pf + num-queue-pairs 3 max-vectors 5 max-mtu 0 rss-key-size 52 rss-lut-size 64 + speed + stats: + rx bytes 40912192 + rx unicast 134856987 + rx discards 94835635 + tx bytes 2442955680 + tx unicast 40715928 + ``` + +- VPP show runtime + + ``` + Thread 1 vpp_wk_0 (lcore 2) + Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6740197.88 + vector rates in 4.2183e3, out 3.7118e3, drop 0.0000e0, punt 0.0000e0 + Name State Calls Vectors Suspends Clocks Vectors/Call + avf-0/3b/2/0-output active 277 34387 0 1.96e1 124.14 + avf-0/3b/2/0-tx active 277 34387 0 3.54e1 124.14 + avf-0/3b/a/0-output active 380 45245 0 1.92e1 119.07 + avf-0/3b/a/0-tx active 380 45245 0 3.36e1 119.07 + avf-input polling 144384995 90499 0 3.03e5 0.00 + ethernet-input active 381 90499 0 1.91e1 237.53 + ip4-input-no-checksum active 381 90499 0 4.94e1 237.53 + ip4-lookup active 521 79632 0 3.76e1 152.84 + ip4-rewrite active 521 79632 0 4.19e1 152.84 + ip4-sv-reassembly-feature active 381 90499 0 3.78e1 237.53 + nat44-ed-in2out active 380 45245 0 1.98e2 119.07 + nat44-ed-in2out-slowpath active 380 45245 0 2.31e3 119.07 + nat44-ed-out2in active 277 34387 0 1.89e2 124.14 + nat44-in2out-worker-handoff active 381 90499 0 9.42e1 237.53 + unix-epoll-input polling 140863 0 0 1.61e3 0.00 + --------------- + Thread 2 vpp_wk_1 (lcore 58) + Time 21.5, 10 sec internal node vector rate 0.00 loops/sec 6733488.17 + vector rates in 3.3365e3, out 3.5604e3, drop 0.0000e0, punt 0.0000e0 + Name State Calls Vectors Suspends Clocks Vectors/Call + avf-0/3b/2/0-output active 276 31129 0 2.03e1 112.79 + avf-0/3b/2/0-tx active 276 31129 0 3.63e1 112.79 + avf-0/3b/a/0-output active 332 45254 0 1.87e1 136.31 + avf-0/3b/a/0-tx active 332 45254 0 3.48e1 136.31 + avf-input polling 166439403 71581 0 4.42e5 0.00 + ethernet-input active 277 65516 0 1.89e1 236.52 + ip4-input-no-checksum active 277 65516 0 4.95e1 236.52 + ip4-lookup active 455 76383 0 3.75e1 167.87 + ip4-rewrite active 455 76383 0 4.20e1 167.87 + ip4-sv-reassembly-feature active 277 65516 0 3.85e1 236.52 + nat44-ed-in2out active 377 45254 0 1.97e2 120.04 + nat44-ed-in2out-slowpath active 332 45254 0 2.39e3 136.31 + nat44-ed-out2in active 276 31129 0 1.83e2 112.79 + nat44-out2in-worker-handoff active 277 65516 0 2.17e2 236.52 + unix-epoll-input polling 140817 0 0 1.60e3 0.00 + ``` + +#### Errors + +- VPP show errors + + ``` + Count Node Reason + 32258 nat44-in2out-worker-handoff same worker + 32256 nat44-in2out-worker-handoff do handoff + 32258 nat44-ed-out2in good out2in packets processed + 32258 nat44-ed-out2in UDP packets + 32258 nat44-ed-in2out-slowpath good in2out packets processed + 32258 nat44-ed-in2out-slowpath UDP packets + 32256 nat44-out2in-worker-handoff same worker + 32258 nat44-out2in-worker-handoff do handoff + 32256 nat44-ed-out2in good out2in packets processed + 32256 nat44-ed-out2in UDP packets + 32256 nat44-ed-in2out-slowpath good in2out packets processed + 32256 nat44-ed-in2out-slowpath UDP packets + ``` + +## TCP/IP CPS Tests + +### TCP/IP TRex Measurements + +#### Counters + +Following TRex ASTF counters are collected by UDP CPS tests for automated +results evaluation (r) and debugging purposes (d): + +- Interface 1 Client + - (d) `opackets` + - (d) `packets` +- Interface 2 Server + - (d) `opackets` + - (d) `packets` +- Traffic Client + - (d) `m_active_flows` + - (d) `m_est_flows` + - (d) `m_traffic_duration` + - (r) `tcps_connattempt` + - (d) `tcps_connects` + - (d) `tcps_closed` +- Traffic Server + - (d) `m_active_flows` + - (d) `m_est_flows` + - (r) `m_traffic_duration` + - (d) `tcps_accepts` + - (r) `tcps_connects` + - (d) `tcps_closed` + - (d) `err_no_template`, server can’t match L7 template no destination port or IP range + +[TRex ASTF counters reference](https://trex-tgn.cisco.com/trex/doc/trex_astf.html#_counters_reference). + +TRex counters are polled only once by CSIT after traffic is stopped. + +#### Calculations + +TODO WIP Note: Currently s_tcp_connects is used for counting successful +sessions. But now I am not sure whether it is correct, as already +c_tcps_connects counts NAT sessions that got established (even though +TCP is not fully connected yet). Not sure how the counters behave when +the third packet is lost and retransmitted. + +- Interface packet loss + - `pktloss_c_s` = `c_opackets` - `s_ipackets` + - `pktloss_s_c` = `s_opackets` - `c_ipackets` + - `pktloss_ratio` = (`pktloss_s_c` + `pktloss_c_s`) / (`c_opackets` + `s_opackets`) +- TCP session integrity + - `tcp_attempted_connection_count` = `c_tcps_connattempt` + - `tcp_failed_connection_count` = `c_tcps_connects` - `c_tcps_connattempt` + +#### CPS Trial PASS + +TODO WIP Note: Currently any trial measurement fails only if TRex itself +fails, or if we fail to parse some counter. No criteria mentioned here +is currently planned to be implemented; we rely on bad things leading to +too few (maybe zero) passed transactions. + +<!-- +PASS of TCP CPS test trial is conditioned on all of the following criteria being met: + +- PASS-C1 TRex must attempt all configured `target_session_number` in `target_setup_duration` time + - IOW TRex must send connect packets at configured `trial_cps_rate`. +- PASS-C2 Following TRex errors ARE NOT recorded in Target-Counters: + - Traffic Client + - No errors recorded so far + - Traffic Server + - `err_no_template`, server can’t match L7 template no destination port or IP range +--> + +#### CPS-MRR + +Reported MRR values are equal to the following TRex counters from Target-Counters: +- `c_m_est_flows` +- `s_m_est_flows` + +TODO Add description of separate set of tests for discovering a **safe** +CPS-MTR value (Maximum Transmit Rate) for TRex, where TRex errors **are not** +observed in Target-Counters. + +#### CPS-PDR + +CPS-PDR values are discovered using MLRsearch, a binary search optimized +for the overall test duration. + +CPS-PDR = `trial_cps_rate`, if all of the following conditions are met: + +- `tcp_failed_connection_count` < `target_loss_ratio` +- `pktloss_ratio` < `target_loss_ratio` + +Measurements to be reported in the CPS-PDR result test message: + +- `trial_cps_rate` +- `c_m_est_flows` +- `s_m_est_flows` + +#### CPS-NDR + +CPS-NDR values are discovered using MLRsearch, a binary search optimized +for the overall test duration. + +CPS-NDR = `trial_cps_rate`, if all of the following conditions are met: + +- `tcp_failed_connection_count` = 0 +- `pktloss_ratio` = 0 + +Measurements to be reported in the CPS-PDR result test message: + +- `trial_cps_rate` +- `c_m_est_flows` +- `s_m_est_flows` + +### TCP/IP VPP Telemetry + +#### Counters + +- VPP show nat44 summary + + ``` + <TODO add sample output> + ``` + +- VPP show interface + + ``` + <TODO add sample output> + ``` + +- VPP show runtime + + ``` + <TODO add sample output> + ``` + +#### Errors + +- VPP show errors + + ``` + <TODO add sample output> + ```
\ No newline at end of file |