Methodology: Edit nat44 test spec doc

- Missing profile specifics and transaction counters.

Change-Id: I6f7378e5fe9d639599c38545b0503354a8a65198
Signed-off-by: Vratko Polak <vrpolak@cisco.com>

1 files changed, 97 insertions, 33 deletions

diff --git a/docs/report/introduction/methodology_nat44.rst b/docs/report/introduction/methodology_nat44.rst
index 5dc558f029..198837d17d 100644
--- a/docs/report/introduction/methodology_nat44.rst
+++ b/docs/report/introduction/methodology_nat44.rst
@@ -12,14 +12,17 @@ public range.
 Following quantities are used to describe inside to outside IP address
 and port bindings scenarios:
 
-- inside-addresses, ports-per-inside-address, number of inside source
-  addresses (representing inside hosts) and number of TCP/UDP source
+- Inside-addresses, number of inside source addresses
+  (representing inside hosts).
+- Ports-per-inside-address, number of TCP/UDP source
   ports per inside source address.
-- outside-addresses, number of outside (public) source addresses
-  allocated to NAT44. The maximal number of ports-per-outside-address
-  usable for NAT is 64 512 (in non-reserved port range 1024-65535,
-  :rfc:`4787`).
-- sharing-ratio, equal to inside-addresses / outside-addresses.
+- Outside-addresses, number of outside (public) source addresses
+  allocated to NAT44.
+- Ports-per-outside-address, number of TCP/UDP source
+  ports per outside source address. The maximal number of
+  ports-per-outside-address usable for NAT is 64 512
+  (in non-reserved port range 1024-65535, :rfc:`4787`).
+- Sharing-ratio, equal to inside-addresses / outside-addresses.
 
 CSIT NAT44 tests are designed to take into account the maximum number of
 ports (sessions) required per inside host (inside-address) and at the
@@ -43,15 +46,7 @@ Initial CSIT NAT44 tests, including associated TG/TRex traffic profiles,
 are based on ports-per-inside-address set to 63 and the sharing ratio of
 1024. This approach is currently used for all NAT44 tests including
 NAT44det (NAT44 deterministic used for Carrier Grade NAT applications)
-and NAT44ed.
-
-..
-    .. TODO::
-
-    Note that in the latter case, due to overloading of (ouside-address,
-    outside-port) tuple for different endpoint destinations the actual
-    sharing ratio is likely to different, as it will depend on the
-    destination addresses used by NAT'ed flows.
+and NAT44ed (Endpoint Dependent).
 
 Private address ranges to be used in tests:
 
@@ -70,13 +65,14 @@ NAT44 Session Scale
 
 NAT44 session scale tested is govern by the following logic:
 
-- Number of inside addresses/hosts H[i] = (H[i-1] x 2^2) with H(0)=1 024, i = 1,2,3, ...
+- Number of inside-addresses(hosts) H[i] = (H[i-1] x 2^2) with H(0)=1 024,
+  i = 1,2,3, ...
 
-  - H[i] = 1 024, 4 096, 16 384, 65 536, 262 144, 1 048 576, ...
+  - H[i] = 1 024, 4 096, 16 384, 65 536, 262 144, ...
 
-- Number of sessions S[i](ports-per-host) = H[i] * ports-per-inside-address
+- Number of sessions S[i] = H[i] * ports-per-inside-address
 
-  - ports-per-host = 63
+  - ports-per-inside-address = 63
 
 +---+---------+------------+
 | i |   hosts |   sessions |
@@ -101,6 +97,11 @@ ip4, ip6 and l2, sending UDP packets in both directions
 inside-to-outside and outside-to-inside. See
 :ref:`data_plane_throughput` for more detail.
 
+The inside-to-outside traffic uses single destination address (20.0.0.0)
+and port (1024).
+The inside-to-outside traffic covers whole inside address and port range,
+the outside-to-inside traffic covers whole outside address and port range.
+
 NAT44det translation entries are created during the ramp-up phase
 preceding the throughput test, followed by verification that all entries
 are present, before proceeding to the throughput test. This ensures
@@ -117,30 +118,55 @@ NAT44det scenario tested:
     16515072.
   - [mrr|ndrpdr|soak], MRR, NDRPDR or SOAK test.
 
+..
+    TODO: The -s{S} part is redundant,
+    we can save space by removing it.
+    TODO: Make traffic profile names resemble suite names more closely.
+
 NAT44 Endpoint-Dependent
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
-NAT44ed is benchmarked using following methodologies:
-
-- Uni-directional throughput using *stateless* traffic profile.
+..
+    TODO: Is it possible to test a NAT44ed scenario where the outside source
+    address and port is limited to just one value?
+    In theory, as long as every inside source address&port traffic
+    uses a different destination address&port, there will be no conflicts,
+    and we could use bidirectional stateless profiles.
+    Possibly, VPP requires some amount of outside source address&port
+    to remain unused for security reasons. But we can try to see what happens.
+
+In order to excercise NAT44ed ability to translate based on both
+source and destination address and port, the inside-to-outside traffic
+varies also destination address and port. Destination port is the same
+as source port, destination address has the same offset as the source address,
+but applied to different subnet (starting with 20.0.0.0).
+
+As the mapping is not deterministic (for security reasons),
+we cannot easily use stateless bidirectional traffic profiles.
+Outside address and port range is fully covered,
+but we do not know which outside-to-inside source address and port to use
+to hit an open session of a particular outside address and port.
+
+Therefore, NAT44ed is benchmarked using following methodologies:
+
+- Unidirectional throughput using *stateless* traffic profile.
 - Connections-per-second using *stateful* traffic profile.
-- Bi-directional throughput using *stateful* traffic profile.
+- Bidirectional PPS (see below) using *stateful* traffic profile.
 
-Uni-directional NAT44ed throughput tests are using TRex STL (Stateless)
+Unidirectional NAT44ed throughput tests are using TRex STL (Stateless)
 APIs and traffic profiles, but with packets sent only in
-inside-to-outside direction. Due to indeterministic bindings of outside
-to inside (src_addr,src_port) that are created dynamically at flow start
-bidirectional testing is not possible with stateless traffic profiles.
-See :ref:`data_plane_throughput` for more detail.
-
-Similarly to NAT44det, NAT44ed uni-directional throughput tests include
+inside-to-outside direction.
+Similarly to NAT44det, NAT44ed unidirectional throughput tests include
 a ramp-up phase to establish and verify the presence of required NAT44ed
-binding entries. NAT44ed CPS (connections-per-second) and throughput /
-PPS stateful tests do not have a ramp-up phase.
+binding entries.
 
 Stateful NAT44ed tests are using TRex ASTF (Advanced Stateful) APIs and
 traffic profiles, with packets sent in both directions. Tests are run
 with both UDP and TCP/IP sessions.
+As both NAT44ed CPS (connections-per-second) and PPS (packets-per-second)
+stateful tests measure (also) session opening performance,
+they use state reset instead of ramp-up trial.
+That is also the reason why PPS tests are not called throughput tests.
 
 Associated CSIT test cases use the following naming scheme to indicate
 NAT44DET case tested:
@@ -164,3 +190,41 @@ NAT44DET case tested:
   - [cps|pps], connections-per-second session establishment rate or
     packets-per-second throughput rate.
   - [mrr|ndrpdr], bidirectional stateful tests MRR, NDRPDR.
+
+Stateful traffic profiles
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+WiP.
+
+UDP CPS
+~~~~~~~
+
+TCP CPS
+~~~~~~~
+
+UDP PPS
+~~~~~~~
+
+TCP PPS
+~~~~~~~
+
+Ip4base tests
+^^^^^^^^^^^^^
+
+Contrary to stateless traffic profiles, we do not have a simple limit
+that would guarantee TRex is able to send traffic at specified load.
+For that reason, we have added tests where "nat44ed" is replaced by "ip4base".
+Instead of NAT44ed processing, the tests set minimalistic IPv4 routes,
+so that packets are forwarded in both inside-to-outside and outside-to-inside
+directions.
+
+The packets arrive to server end of TRex with different source address&port
+than in NAT44ed tests (no translation to outside values is done with ip4base),
+but those are not specified in the stateful traffic profiles.
+The server end uses the received address&port as destination
+for outside-to-inside traffic. Therefore the same stateful traffic profile
+works for both NAT44ed and ip4base test (of the same scale).
+
+The NAT44ed results are displayed together with corresponding ip4base results.
+If they are similar, TRex is probably the bottleneck.
+If NAT44ed result is visibly smaller, it describes the real VPP performance.