aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authorMaciek Konstantynowicz <mkonstan@cisco.com>2018-11-07 11:17:49 +0000
committerMaciek Konstantynowicz <mkonstan@cisco.com>2018-11-07 12:09:32 +0000
commit90809c1375e79d4a3220f995bd0ed3b7762e7df8 (patch)
tree33d29902344bd04732856e491cf7e214334d7a52
parente0806c8358f38406432d1c6f09de7b90fadb8646 (diff)
report: test methodology updated with new bits and bites in 18.10.
Change-Id: I08fcb7c6d09adc37805083319d148c9659a80eff Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
-rw-r--r--docs/report/introduction/introduction.rst3
-rw-r--r--docs/report/introduction/methodology.rst81
-rw-r--r--docs/report/vpp_performance_tests/test_environment.rst2
3 files changed, 60 insertions, 26 deletions
diff --git a/docs/report/introduction/introduction.rst b/docs/report/introduction/introduction.rst
index 24c5970725..9990d47d5d 100644
--- a/docs/report/introduction/introduction.rst
+++ b/docs/report/introduction/introduction.rst
@@ -13,7 +13,8 @@ available for download.
- **Test Scenarios Overview**: A brief overview of test scenarios
covered in this report.
- **Physical Testbeds**: Description of physical testbeds.
- - **Performance Test Methodology**: Benchmarking test methodologies.
+ - **Test Methodology**: Performance benchmarking and functional test
+ methodologies.
#. VPP PERFORMANCE: VPP performance tests executed in physical
FD.io testbeds.
diff --git a/docs/report/introduction/methodology.rst b/docs/report/introduction/methodology.rst
index 89db393973..f6217d89a6 100644
--- a/docs/report/introduction/methodology.rst
+++ b/docs/report/introduction/methodology.rst
@@ -1,16 +1,17 @@
-.. _performance_test_methodology:
+.. _test_methodology:
-Performance Test Methodology
-============================
+Test Methodology
+================
-Throughput
-----------
+Data Plane Throughput
+---------------------
-Packet and bandwidth throughput are measured in accordance with
-:rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search (MLRsearch), an
-optimized binary search algorithm, that measures SUT/DUT throughput at
-different Packet Loss Ratio (PLR) values.
+Network data plane packet and bandwidth throughput are measured in
+accordance with :rfc:`2544`, using FD.io CSIT Multiple Loss Ratio search
+(MLRsearch), an optimized throughput search algorithm, that measures
+SUT/DUT packet throughput rates at different Packet Loss Ratio (PLR)
+values.
Following MLRsearch values are measured across a range of L2 frame sizes
and reported:
@@ -39,13 +40,15 @@ All rates are reported from external Traffic Generator perspective.
.. _mlrsearch_algorithm:
-MLRsearch Algorithm
--------------------
+MLRsearch Tests
+---------------
-Multiple Loss Rate search (MLRsearch) is a new search algorithm
+Multiple Loss Rate search (MLRsearch) tests use new search algorithm
implemented in FD.io CSIT project. MLRsearch discovers multiple packet
throughput rates in a single search, with each rate associated with a
-distinct Packet Loss Ratio (PLR) criteria.
+distinct Packet Loss Ratio (PLR) criteria. MLRsearch is being
+standardized in IETF with `draft-vpolak-mkonstan-mlrsearch-XX
+<https://tools.ietf.org/html/draft-vpolak-mkonstan-mlrsearch-00>`_.
Two throughput measurements used in FD.io CSIT are Non-Drop Rate (NDR,
with zero packet loss, PLR=0) and Partial Drop Rate (PDR, with packet
@@ -318,15 +321,22 @@ but without detailing their mutual interaction.
than pure binary search, the implemented tests fail themselves
when the search takes too long (given by argument *timeout*).
-Maximum Receive Rate MRR
-------------------------
+(B)MRR Throughput
+-----------------
-MRR tests measure the packet forwarding rate under the maximum
-load offered by traffic generator over a set trial duration,
+Maximum Receive Rate (MRR) tests are complementary to MLRsearch tests,
+as they provide a maximum "raw" throughput benchmark for development and
+testing community. MRR tests measure the packet forwarding rate under
+the maximum load offered by traffic generator over a set trial duration,
regardless of packet loss. Maximum load for specified Ethernet frame
size is set to the bi-directional link rate.
-Current parameters for MRR tests:
+In |csit-release| MRR test code has been updated with a configurable
+burst MRR parameters: trial duration and number of trials in a single
+burst. This enabled a new Burst MRR (BMRR) methodology for more precise
+performance trending.
+
+Current parameters for BMRR tests:
- Ethernet frame sizes: 64B (78B for IPv6), IMIX, 1518B, 9000B; all
quoted sizes include frame CRC, but exclude per frame transmission
@@ -346,17 +356,25 @@ Current parameters for MRR tests:
XL710. Packet rate for other tested frame sizes is limited by PCIe
Gen3 x8 bandwidth limitation of ~50Gbps.
-- Trial duration: 10sec.
+- Trial duration: 1 sec.
+
+- Number of trials per burst: 10.
Similarly to NDR/PDR throughput tests, MRR test should be reporting bi-
directional link rate (or NIC rate, if lower) if tested VPP
configuration can handle the packet rate higher than bi-directional link
rate, e.g. large packet tests and/or multi-core tests.
-MRR tests are used for continuous performance trending and for
-comparison between releases. Daily trending job tests subset of frame
-sizes, focusing on 64B (78B for IPv6) for all tests and IMIX for
-selected tests (vhost, memif).
+MRR tests are currently used for FD.io CSIT continuous performance
+trending and for comparison between releases. Daily trending job tests
+subset of frame sizes, focusing on 64B (78B for IPv6) for all tests and
+IMIX for selected tests (vhost, memif).
+
+MRR-like measurements are being used to establish starting conditions
+for experimental Probabilistic Loss Ratio Search (PLRsearch) used for
+soak testing, aimed at verifying continuous system performance over an
+extended period of time, hours, days, weeks, months. PLRsearch code is
+currently in experimental phase in FD.io CSIT project.
Packet Latency
--------------
@@ -510,7 +528,8 @@ following environment settings:
[cfs,cfsrr1] settings.
- Adjusted Linux kernel :abbr:`CFS (Completely Fair Scheduler)`
scheduler policy for data plane threads used in CSIT is documented in
- `CSIT Performance Environment Tuning wiki <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_.
+ `CSIT Performance Environment Tuning wiki
+ <https://wiki.fd.io/view/CSIT/csit-perf-env-tuning-ubuntu1604>`_.
- The purpose is to verify performance impact (MRR and NDR/PDR
throughput) and same test measurements repeatability, by making VPP
and VM data plane threads less susceptible to other Linux OS system
@@ -562,6 +581,20 @@ VMs as described earlier in :ref:`tested_physical_topologies`.
Further documentation is available in
:ref:`container_orchestration_in_csit`.
+VPP_Device Functional
+---------------------
+
+|csit-release| added new VPP_Device test environment for functional VPP
+device tests integrated into LFN CI/CD infrastructure. VPP_Device tests
+run on 1-Node testbeds (1n-skx, 1n-arm) and rely on Linux SRIOV Virtual
+Function (VF), dot1q VLAN tagging and external loopback cables to
+facilitate packet passing over exernal physical links. Initial focus is
+on few baseline tests. Existing CSIT VIRL tests can be moved to
+VPP_Device framework by changing L1 and L2 KW(s). RF test definition
+code stays unchanged with the exception of requiring adjustments from
+3-Node to 2-Node logical topologies. CSIT VIRL to VPP_Device migration
+is expected in the next CSIT release.
+
IPSec on Intel QAT
------------------
diff --git a/docs/report/vpp_performance_tests/test_environment.rst b/docs/report/vpp_performance_tests/test_environment.rst
index 6dd6ca35d8..8f92c6ef20 100644
--- a/docs/report/vpp_performance_tests/test_environment.rst
+++ b/docs/report/vpp_performance_tests/test_environment.rst
@@ -103,6 +103,6 @@ below:
}
Description of VPP startup settings used in CSIT is provided in
-:ref:`performance_test_methodology`.
+:ref:`test_methodology`.
.. include:: ../introduction/test_environment_tg.rst