aboutsummaryrefslogtreecommitdiffstats
path: root/docs/report/honeycomb_performance_tests
diff options
context:
space:
mode:
authorselias <samelias@cisco.com>2017-07-20 09:50:58 +0200
committerselias <samelias@cisco.com>2017-07-24 15:22:02 +0200
commitcd8d4dc1498a69b4f9541fb4405fd31fd61d81d9 (patch)
tree2a00205af6500a085afcb8306beec29ab5d1b584 /docs/report/honeycomb_performance_tests
parent1911067935bdf6481fc9b89f40d79ca61f2448d0 (diff)
Static report content - Honeycomb
Change-Id: I39899747f99684b932bb9c88268391e55b6fc779 Signed-off-by: selias <samelias@cisco.com>
Diffstat (limited to 'docs/report/honeycomb_performance_tests')
-rw-r--r--docs/report/honeycomb_performance_tests/csit_release_notes.rst19
-rw-r--r--docs/report/honeycomb_performance_tests/documentation.rst5
-rw-r--r--docs/report/honeycomb_performance_tests/index.rst11
-rw-r--r--docs/report/honeycomb_performance_tests/overview.rst124
-rw-r--r--docs/report/honeycomb_performance_tests/test_environment.rst22
-rw-r--r--docs/report/honeycomb_performance_tests/test_result_data.rst101
6 files changed, 282 insertions, 0 deletions
diff --git a/docs/report/honeycomb_performance_tests/csit_release_notes.rst b/docs/report/honeycomb_performance_tests/csit_release_notes.rst
new file mode 100644
index 0000000000..51b62a7a6a
--- /dev/null
+++ b/docs/report/honeycomb_performance_tests/csit_release_notes.rst
@@ -0,0 +1,19 @@
+CSIT Release Notes
+==================
+
+Changes in CSIT |release|
+-------------------------
+
+#. First release with honeycomb performance testing
+
+Known Issues
+------------
+
+Here is the list of known issues in CSIT |release| for Honeycomb performance
+tests in VIRL:
+
++---+--------------------------------------------+------------+----------------------------------------------------------------------------+
+| # | Issue | Jira ID | Description |
++---+--------------------------------------------+------------+----------------------------------------------------------------------------+
+| 1 | | | |
++---+--------------------------------------------+------------+----------------------------------------------------------------------------+
diff --git a/docs/report/honeycomb_performance_tests/documentation.rst b/docs/report/honeycomb_performance_tests/documentation.rst
new file mode 100644
index 0000000000..6b15bde6ee
--- /dev/null
+++ b/docs/report/honeycomb_performance_tests/documentation.rst
@@ -0,0 +1,5 @@
+Documentation
+=============
+
+`CSIT Honeycomb Performance Tests Documentation`_ contains detailed
+functional description and input parameters for each test case.
diff --git a/docs/report/honeycomb_performance_tests/index.rst b/docs/report/honeycomb_performance_tests/index.rst
new file mode 100644
index 0000000000..3177494395
--- /dev/null
+++ b/docs/report/honeycomb_performance_tests/index.rst
@@ -0,0 +1,11 @@
+Honeycomb Performance Tests
+===========================
+
+.. toctree::
+
+ overview
+ csit_release_notes
+ test_environment
+ documentation
+ test_result_data
+
diff --git a/docs/report/honeycomb_performance_tests/overview.rst b/docs/report/honeycomb_performance_tests/overview.rst
new file mode 100644
index 0000000000..0b2e3c41b8
--- /dev/null
+++ b/docs/report/honeycomb_performance_tests/overview.rst
@@ -0,0 +1,124 @@
+Overview
+========
+
+Tested Physical Topologies
+--------------------------
+
+CSIT VPP performance tests are executed on physical baremetal servers hosted by
+LF FD.io project. Testbed physical topology is shown in the figure below.
+
+::
+
+ +------------------------+ +------------------------+
+ | | | |
+ | +------------------+ | | +------------------+ |
+ | | | | | | | |
+ | | <-----------------> | |
+ | | DUT1 | | | | DUT2 | |
+ | +--^---------------+ | | +---------------^--+ |
+ | | | | | |
+ | | SUT1 | | SUT2 | |
+ +------------------------+ +------------------^-----+
+ | |
+ | |
+ | +-----------+ |
+ | | | |
+ +------------------> TG <------------------+
+ | |
+ +-----------+
+
+SUT1 runs VPP SW application in Linux user-mode as a
+Device Under Test (DUT), and a python script to generate traffic. SUT2 and TG
+are unused.
+sical connectivity between SUTs and to TG is provided using
+different NIC model. Currently installed NIC models include:
+
+Performance tests involve sending Netconf requests over localhost to the
+Honeycomb listener port, and measuring response time.
+
+Note that reported performance results are specific to the SUTs tested.
+Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with
+other CPUs are likely to yield different results.
+
+For detailed LF FD.io test bed specification and physical topology please refer
+to `LF FDio CSIT testbed wiki page
+<https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.
+
+Performance Tests Coverage
+--------------------------
+
+As of right now, there is only a single Honeycomb performance test. Measuring
+response time for a simple read operation, performed synchronously and using
+single (not batch) requests.
+
+Currently the tests do not trigger automatically, but can be run on-demand from
+the hc2vpp project.
+
+Performance Tests Naming
+------------------------
+
+CSIT |release| follows a common structured naming convention for all
+performance and system functional tests, introduced in CSIT |release-1|.
+
+The naming should be intuitive for majority of the tests. Complete
+description of CSIT test naming convention is provided on `CSIT test naming wiki
+<https://wiki.fd.io/view/CSIT/csit-test-naming>`_.
+
+Here few illustrative examples of the new naming usage for performance test
+suites:
+
+#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P**
+
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType*
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on
+ Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching
+ with MAC learning, NDR throughput discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE
+ on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline
+ switching with MAC learning, NDR throughput discovery.
+ - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel
+ x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery.
+ - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on
+ Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput
+ discovery.
+
+#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC,
+ P2V2P, NIC2VMchain2NIC, P2V2V2P**
+
+ - *PortNICConfig-WireEncapsulation-PacketForwardingFunction-
+ PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation-
+ VirtPortConfig-VMconfig-TestType*
+ - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports
+ of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain
+ switching to/from two vhost interfaces and one VM, NDR throughput
+ discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from two vhost interfaces and one VM, NDR throughput
+ discovery.
+ - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2
+ ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain
+ switching to/from four vhost interfaces and two VMs, NDR throughput
+ discovery.
+
+Methodology: Multi-Core
+-----------------------
+
+**Multi-core Test** - CSIT |release| multi-core tests are executed in the
+following thread and core configurations:
+
+#. 1t - 1 Honeycomb Netconf thread on 1 CPU physical core.
+#. 8t - 8 Honeycomb Netconf thread on 8 CPU physical core.
+#. 16t - 16 Honeycomb Netconf thread on 16 CPU physical core.
+
+Traffic generator also uses multiple threads/cores, to simulate multiple
+Netconf clients accessing the Honeycomb server.
+
+Methodology: Performance measurement
+------------------------------------
+
+The following values are measured and reported in tests:
+
+- Average request rate. Averaged over the entire test duration, over all client
+ threads. Negative replies (if any) are not counted and are reported separately.
diff --git a/docs/report/honeycomb_performance_tests/test_environment.rst b/docs/report/honeycomb_performance_tests/test_environment.rst
new file mode 100644
index 0000000000..1cafe26aa4
--- /dev/null
+++ b/docs/report/honeycomb_performance_tests/test_environment.rst
@@ -0,0 +1,22 @@
+Test Environment
+================
+
+To execute performance tests, there are three identical testbeds, each testbed
+consists of two SUTs and one TG.
+
+Server HW Configuration
+-----------------------
+
+See `Performance HW Configuration <../vpp_performance_tests/test_environment.html>`_
+
+Additionally, configuration for the Honeycomb client:
+
+
+**Honeycomb Startup Command**
+
+Use the server mode JIT compiler, increase the default memory size,
+metaspace size, and enable NUMA optimizations for the JVM.
+
+::
+
+ $ java -server -Xms128m -Xmx512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m -XX:+UseNUMA -XX:+UseParallelGC
diff --git a/docs/report/honeycomb_performance_tests/test_result_data.rst b/docs/report/honeycomb_performance_tests/test_result_data.rst
new file mode 100644
index 0000000000..563e93ea5f
--- /dev/null
+++ b/docs/report/honeycomb_performance_tests/test_result_data.rst
@@ -0,0 +1,101 @@
+Test Result Data
+================
+
+This section includes summary of Netconf read operation performance.
+Performance is reported for Honeycomb running in multiple configurations of
+netconf thread(s) and their physical CPU core(s) placement, and for different
+read operation targets.
+
+.. note::
+
+ Test results have been generated by
+ `FD.io test executor honeycomb performance jobs`_ with Robot Framework
+ result files csit-vpp-perf-\*.zip `archived here <../../_static/archive/>`_.
+
+Honeycomb + Netconf
+===================
+
+Performs read operations from Honeycomb's operational data store. Honeycomb
+does not contact VPP to obtain the most up-to-date value. Operations are
+performed synchronously per client, varying the number of clients from 1 to 16.
+
+netconf-netty-threads: 16
+
++----------------+----------------------------------+----------------+
+| # clients | TCP performance (reads/sec) | Total requests |
++================+==================================+================+
+| 1 | 6630 | 100K |
++----------------+----------------------------------+----------------+
+| 2 | 14598 | 100K |
++----------------+----------------------------------+----------------+
+| 4 | 28309 | 100K |
++----------------+----------------------------------+----------------+
+| 8 | 46715 | 100K |
++----------------+----------------------------------+----------------+
+| 16 | 458141 | 100K |
++----------------+----------------------------------+----------------+
+
+netconf-netty-threads: 1
+
++----------------+----------------------------------+----------------+
+| # clients | TCP performance (reads/sec) | Total requests |
++================+==================================+================+
+| 1 | 6563 | 100K |
++----------------+----------------------------------+----------------+
+| 2 | 7601 | 100K |
++----------------+----------------------------------+----------------+
+| 4 | 8212 | 100K |
++----------------+----------------------------------+----------------+
+| 8 | 8595 | 100K |
++----------------+----------------------------------+----------------+
+| 16 | 8699 | 100K |
++----------------+----------------------------------+----------------+
+
+Data source:
+https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-perf-master-ubuntu1604/4/
+
+Note: At 46K/s we are likely hitting the limits of the Netconf interface,
+according to https://wiki.opendaylight.org/view/NETCONF:Testing#Results_3
+
+Honeycomb + Netconf + VPP
+=========================
+
+Performs read operations from Honeycomb's operational data store. Honeycomb
+uses VPP API to obtain the value from VPP before responding. Operations are
+performed synchronously per client, varying the number of clients from 1 to 16.
+
+netconf-netty-threads: 16
+# Results pending
+
++----------------+----------------------------------+----------------+
+| # clients | TCP performance (reads/sec) | Total requests |
++================+==================================+================+
+| 1 | | 100K |
++----------------+----------------------------------+----------------+
+| 2 | | 100K |
++----------------+----------------------------------+----------------+
+| 4 | | 100K |
++----------------+----------------------------------+----------------+
+| 8 | | 100K |
++----------------+----------------------------------+----------------+
+| 16 | | 100K |
++----------------+----------------------------------+----------------+
+
+netconf-netty-threads: 1
+
++----------------+----------------------------------+----------------+
+| # clients | TCP performance (reads/sec) | Total requests |
++================+==================================+================+
+| 1 | | 100K |
++----------------+----------------------------------+----------------+
+| 2 | | 100K |
++----------------+----------------------------------+----------------+
+| 4 | | 100K |
++----------------+----------------------------------+----------------+
+| 8 | | 100K |
++----------------+----------------------------------+----------------+
+| 16 | | 100K |
++----------------+----------------------------------+----------------+
+
+Data source:
+# TODO