diff options
Diffstat (limited to 'docs/report/honeycomb_performance_tests')
6 files changed, 282 insertions, 0 deletions
diff --git a/docs/report/honeycomb_performance_tests/csit_release_notes.rst b/docs/report/honeycomb_performance_tests/csit_release_notes.rst new file mode 100644 index 0000000000..51b62a7a6a --- /dev/null +++ b/docs/report/honeycomb_performance_tests/csit_release_notes.rst @@ -0,0 +1,19 @@ +CSIT Release Notes
+==================
+
+Changes in CSIT |release|
+-------------------------
+
+#. First release with honeycomb performance testing
+
+Known Issues
+------------
+
+Here is the list of known issues in CSIT |release| for Honeycomb performance
+tests in VIRL:
+
++---+--------------------------------------------+------------+----------------------------------------------------------------------------+
+| # | Issue | Jira ID | Description |
++---+--------------------------------------------+------------+----------------------------------------------------------------------------+
+| 1 | | | |
++---+--------------------------------------------+------------+----------------------------------------------------------------------------+
diff --git a/docs/report/honeycomb_performance_tests/documentation.rst b/docs/report/honeycomb_performance_tests/documentation.rst new file mode 100644 index 0000000000..6b15bde6ee --- /dev/null +++ b/docs/report/honeycomb_performance_tests/documentation.rst @@ -0,0 +1,5 @@ +Documentation
+=============
+
+`CSIT Honeycomb Performance Tests Documentation`_ contains detailed
+functional description and input parameters for each test case.
diff --git a/docs/report/honeycomb_performance_tests/index.rst b/docs/report/honeycomb_performance_tests/index.rst new file mode 100644 index 0000000000..3177494395 --- /dev/null +++ b/docs/report/honeycomb_performance_tests/index.rst @@ -0,0 +1,11 @@ +Honeycomb Performance Tests +=========================== + +.. toctree:: + + overview + csit_release_notes + test_environment + documentation + test_result_data + diff --git a/docs/report/honeycomb_performance_tests/overview.rst b/docs/report/honeycomb_performance_tests/overview.rst new file mode 100644 index 0000000000..0b2e3c41b8 --- /dev/null +++ b/docs/report/honeycomb_performance_tests/overview.rst @@ -0,0 +1,124 @@ +Overview +======== + +Tested Physical Topologies +-------------------------- + +CSIT VPP performance tests are executed on physical baremetal servers hosted by +LF FD.io project. Testbed physical topology is shown in the figure below. + +:: + + +------------------------+ +------------------------+ + | | | | + | +------------------+ | | +------------------+ | + | | | | | | | | + | | <-----------------> | | + | | DUT1 | | | | DUT2 | | + | +--^---------------+ | | +---------------^--+ | + | | | | | | + | | SUT1 | | SUT2 | | + +------------------------+ +------------------^-----+ + | | + | | + | +-----------+ | + | | | | + +------------------> TG <------------------+ + | | + +-----------+ + +SUT1 runs VPP SW application in Linux user-mode as a +Device Under Test (DUT), and a python script to generate traffic. SUT2 and TG +are unused. +sical connectivity between SUTs and to TG is provided using +different NIC model. Currently installed NIC models include: + +Performance tests involve sending Netconf requests over localhost to the +Honeycomb listener port, and measuring response time. + +Note that reported performance results are specific to the SUTs tested. +Current LF FD.io SUTs are based on Intel XEON E5-2699v3 2.3GHz CPUs. SUTs with +other CPUs are likely to yield different results. + +For detailed LF FD.io test bed specification and physical topology please refer +to `LF FDio CSIT testbed wiki page +<https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_. + +Performance Tests Coverage +-------------------------- + +As of right now, there is only a single Honeycomb performance test. Measuring +response time for a simple read operation, performed synchronously and using +single (not batch) requests. + +Currently the tests do not trigger automatically, but can be run on-demand from +the hc2vpp project. + +Performance Tests Naming +------------------------ + +CSIT |release| follows a common structured naming convention for all +performance and system functional tests, introduced in CSIT |release-1|. + +The naming should be intuitive for majority of the tests. Complete +description of CSIT test naming convention is provided on `CSIT test naming wiki +<https://wiki.fd.io/view/CSIT/csit-test-naming>`_. + +Here few illustrative examples of the new naming usage for performance test +suites: + +#. **Physical port to physical port - a.k.a. NIC-to-NIC, Phy-to-Phy, P2P** + + - *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-TestType* + - *10ge2p1x520-dot1q-l2bdbasemaclrn-ndrdisc.robot* => 2 ports of 10GE on + Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain baseline switching + with MAC learning, NDR throughput discovery. + - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-ndrchk.robot* => 2 ports of 10GE + on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain baseline + switching with MAC learning, NDR throughput discovery. + - *10ge2p1x520-ethip4-ip4base-ndrdisc.robot* => 2 ports of 10GE on Intel + x520 NIC, IPv4 baseline routed forwarding, NDR throughput discovery. + - *10ge2p1x520-ethip6-ip6scale200k-ndrdisc.robot* => 2 ports of 10GE on + Intel x520 NIC, IPv6 scaled up routed forwarding, NDR throughput + discovery. + +#. **Physical port to VM (or VM chain) to physical port - a.k.a. NIC2VM2NIC, + P2V2P, NIC2VMchain2NIC, P2V2V2P** + + - *PortNICConfig-WireEncapsulation-PacketForwardingFunction- + PacketProcessingFunction1-...-PacketProcessingFunctionN-VirtEncapsulation- + VirtPortConfig-VMconfig-TestType* + - *10ge2p1x520-dot1q-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 ports + of 10GE on Intel x520 NIC, dot1q tagged Ethernet, L2 bridge-domain + switching to/from two vhost interfaces and one VM, NDR throughput + discovery. + - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-2vhost-1vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from two vhost interfaces and one VM, NDR throughput + discovery. + - *10ge2p1x520-ethip4vxlan-l2bdbasemaclrn-eth-4vhost-2vm-ndrdisc.robot* => 2 + ports of 10GE on Intel x520 NIC, IPv4 VXLAN Ethernet, L2 bridge-domain + switching to/from four vhost interfaces and two VMs, NDR throughput + discovery. + +Methodology: Multi-Core +----------------------- + +**Multi-core Test** - CSIT |release| multi-core tests are executed in the +following thread and core configurations: + +#. 1t - 1 Honeycomb Netconf thread on 1 CPU physical core. +#. 8t - 8 Honeycomb Netconf thread on 8 CPU physical core. +#. 16t - 16 Honeycomb Netconf thread on 16 CPU physical core. + +Traffic generator also uses multiple threads/cores, to simulate multiple +Netconf clients accessing the Honeycomb server. + +Methodology: Performance measurement +------------------------------------ + +The following values are measured and reported in tests: + +- Average request rate. Averaged over the entire test duration, over all client + threads. Negative replies (if any) are not counted and are reported separately. diff --git a/docs/report/honeycomb_performance_tests/test_environment.rst b/docs/report/honeycomb_performance_tests/test_environment.rst new file mode 100644 index 0000000000..1cafe26aa4 --- /dev/null +++ b/docs/report/honeycomb_performance_tests/test_environment.rst @@ -0,0 +1,22 @@ +Test Environment +================ + +To execute performance tests, there are three identical testbeds, each testbed +consists of two SUTs and one TG. + +Server HW Configuration +----------------------- + +See `Performance HW Configuration <../vpp_performance_tests/test_environment.html>`_ + +Additionally, configuration for the Honeycomb client: + + +**Honeycomb Startup Command** + +Use the server mode JIT compiler, increase the default memory size, +metaspace size, and enable NUMA optimizations for the JVM. + +:: + + $ java -server -Xms128m -Xmx512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m -XX:+UseNUMA -XX:+UseParallelGC diff --git a/docs/report/honeycomb_performance_tests/test_result_data.rst b/docs/report/honeycomb_performance_tests/test_result_data.rst new file mode 100644 index 0000000000..563e93ea5f --- /dev/null +++ b/docs/report/honeycomb_performance_tests/test_result_data.rst @@ -0,0 +1,101 @@ +Test Result Data +================ + +This section includes summary of Netconf read operation performance. +Performance is reported for Honeycomb running in multiple configurations of +netconf thread(s) and their physical CPU core(s) placement, and for different +read operation targets. + +.. note:: + + Test results have been generated by + `FD.io test executor honeycomb performance jobs`_ with Robot Framework + result files csit-vpp-perf-\*.zip `archived here <../../_static/archive/>`_. + +Honeycomb + Netconf +=================== + +Performs read operations from Honeycomb's operational data store. Honeycomb +does not contact VPP to obtain the most up-to-date value. Operations are +performed synchronously per client, varying the number of clients from 1 to 16. + +netconf-netty-threads: 16 + ++----------------+----------------------------------+----------------+ +| # clients | TCP performance (reads/sec) | Total requests | ++================+==================================+================+ +| 1 | 6630 | 100K | ++----------------+----------------------------------+----------------+ +| 2 | 14598 | 100K | ++----------------+----------------------------------+----------------+ +| 4 | 28309 | 100K | ++----------------+----------------------------------+----------------+ +| 8 | 46715 | 100K | ++----------------+----------------------------------+----------------+ +| 16 | 458141 | 100K | ++----------------+----------------------------------+----------------+ + +netconf-netty-threads: 1 + ++----------------+----------------------------------+----------------+ +| # clients | TCP performance (reads/sec) | Total requests | ++================+==================================+================+ +| 1 | 6563 | 100K | ++----------------+----------------------------------+----------------+ +| 2 | 7601 | 100K | ++----------------+----------------------------------+----------------+ +| 4 | 8212 | 100K | ++----------------+----------------------------------+----------------+ +| 8 | 8595 | 100K | ++----------------+----------------------------------+----------------+ +| 16 | 8699 | 100K | ++----------------+----------------------------------+----------------+ + +Data source: +https://jenkins.fd.io/view/hc2vpp/job/hc2vpp-csit-perf-master-ubuntu1604/4/ + +Note: At 46K/s we are likely hitting the limits of the Netconf interface, +according to https://wiki.opendaylight.org/view/NETCONF:Testing#Results_3 + +Honeycomb + Netconf + VPP +========================= + +Performs read operations from Honeycomb's operational data store. Honeycomb +uses VPP API to obtain the value from VPP before responding. Operations are +performed synchronously per client, varying the number of clients from 1 to 16. + +netconf-netty-threads: 16 +# Results pending + ++----------------+----------------------------------+----------------+ +| # clients | TCP performance (reads/sec) | Total requests | ++================+==================================+================+ +| 1 | | 100K | ++----------------+----------------------------------+----------------+ +| 2 | | 100K | ++----------------+----------------------------------+----------------+ +| 4 | | 100K | ++----------------+----------------------------------+----------------+ +| 8 | | 100K | ++----------------+----------------------------------+----------------+ +| 16 | | 100K | ++----------------+----------------------------------+----------------+ + +netconf-netty-threads: 1 + ++----------------+----------------------------------+----------------+ +| # clients | TCP performance (reads/sec) | Total requests | ++================+==================================+================+ +| 1 | | 100K | ++----------------+----------------------------------+----------------+ +| 2 | | 100K | ++----------------+----------------------------------+----------------+ +| 4 | | 100K | ++----------------+----------------------------------+----------------+ +| 8 | | 100K | ++----------------+----------------------------------+----------------+ +| 16 | | 100K | ++----------------+----------------------------------+----------------+ + +Data source: +# TODO |