diff options
Diffstat (limited to 'resources/tools/presentation_new/doc/graphs_improvements.rst')
| -rw-r--r-- | resources/tools/presentation_new/doc/graphs_improvements.rst | 637 |
1 files changed, 0 insertions, 637 deletions
diff --git a/resources/tools/presentation_new/doc/graphs_improvements.rst b/resources/tools/presentation_new/doc/graphs_improvements.rst deleted file mode 100644 index 336faf9748..0000000000 --- a/resources/tools/presentation_new/doc/graphs_improvements.rst +++ /dev/null @@ -1,637 +0,0 @@ -================================ - Envisioning information by PAL -================================ - -Introduction ------------- - -This document describes possible improvements in data presentation provided by -PAL for the `Report <https://docs.fd.io/csit/master/report/>`_ and the -`Trending <https://docs.fd.io/csit/master/trending/>`_ - -You can generate a standalone html version of this document using e.g. -rst2html5 tool: - -.. code:: bash - - rst2html5 --stylesheet graphs_improvements.css graphs_improvements.rst >> graphs_improvements.html - -**Modifications of existing graphs** - -- `Speedup Multi-core`_ -- `Packet Throughput`_ -- `Packet Latency`_ -- `HTTP-TCP Performance`_ - -**New graphs to be added** - -- `Comparison between releases`_ -- `Comparison between processor architectures`_ -- `Comparison between 2-node and 3-node topologies`_ -- `Comparison between different physical testbed instances`_ -- `Comparison between NICs`_ -- `Other comparisons`_ - -**Export of static images** - -- low priority -- make possible to `export static images`_ which are available via link on the - web page. -- vector formats (svg, pdf) are preferred - -Priorities ----------- - -**Target CSIT-18.10** - -- `Speedup Multi-core`_ -- `Packet Throughput`_ - -**Nice to have in CSIT-18.10** - -.. note:: - - Only if above done, and target CSIT-18.10 is in time , otherwise next - release. - -- `Packet Latency`_ -- `HTTP-TCP Performance`_ - -Modifications of existing graphs --------------------------------- - -The proposed modifications include the changes in: - -- the layout of the graphs, -- the data and way how it is presented, -- the test cases presented in the graphs. - -The first two points are described below, the last one will be added later as a -separate chapter. - -.. _Speedup Multi-core: - -Speedup Multi-core -`````````````````` - -The "Speedup Multicore" graph will display the measured data together with -perfect values calculated as multiples of the best value measured using one -core. The relative difference between measured and perfect values will be -displayed in the hover next to each data point. - -.. image:: pic/graph-speedup.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Speedup Multi-core: not found. - -**Description:** - -*Data displayed:* - -- one or more data series from the same area, keep the number of displayed - data series as low as possible (max 6) -- x-axis: number of cores -- y-axis: throughput (measured and perfect) [Mpps], linear scale, beginning - with 0 -- hover information: Throughput [Mpps], Speedup [1], Relative difference between - measured and ideal values [%], Perfect Throughput [%] -- Limits of ethernet links, NICs and PCIe. See `Physical performance limits`_. - -*Layout:* - -- plot type: lines with data points (plotly.graph_objs.Scatter) -- data series format: - - measured: solid line with data points - - perfect: dashed line with data points, the same color as "measured" -- title: "Speedup Multi-core: <area, scaling, features, ...>", - top, centered, font size 18; configurable in specification file: visible / - hidden, text -- x-axis: integers, starting with 1 (core), linear, font size 16, bottom -- x-axis label: "Number of cores [qty]", bottom, centered, font size 16 -- y-axis: float, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Packet Throughput [Mpps]", middle, left -- legend: list of presented test cases, bottom, left, font size 16; the order - of displayed tests is configurable in the specification file -- annotation: text: "dashed: perfect<br>solid: measured", top, left, - font size 16 - -.. _Packet Throughput: - -Packet Throughput -````````````````` - -The "Packet Throughput" graph will display the measured data using -statistical box graph. Each data point is constructed from 10 samples. -The statistical data are displayed as hover information. - -.. image:: pic/graph-throughput.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Packet Throughput" not found. - -**Description:** - -*Data displayed:* - -- one or more data points from the same area, keep the number of displayed - data points as low as possible (max 6) -- x-axis: indexed test cases -- y-axis: throughput [Mpps], linear scale, beginning with 0 -- hover information: statistical data (min, lower fence, q1, median, q3, - higher fence, max), test case name - -*Layout:* - -- plot type: statistical box (plotly.graph_objs.Box) -- data series format: box -- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>", - top, centered, font size 18; configurable in specification file: visible / - hidden, text -- x-axis: integers, starting with 1, linear, font size 16, bottom; the order - of displayed tests is configurable in the specification file -- x-axis label: "Indices of Test Cases [Index]", bottom, centered, font size 16 -- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Packet Throughput [Mpps]", middle, left -- legend: "Indexed Test Cases [idx]", bottom, left, font size 16 - -.. _Packet Latency: - -Packet Latency -`````````````` - -The "Packet Latency" graph will display the measured data using -statistical box graph. Each data point is constructed from 10 samples. -The statistical data are displayed as hover information. - -.. image:: pic/graph-latency.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Packet Latency" not found. - -**Description:** - -*Data displayed:* - -- one or more data points from the same area, keep the number of displayed - data points as low as possible (max 6) -- x-axis: data flow directions -- y-axis: latency min/avg/max [uSec], linear scale, beginning with 0 -- hover information: statistical data (min, avg, max), test case name, direction - -*Layout:* - -- plot type: scatter with errors (plotly.graph_objs.Scatter) -- data series format: data point with min amd max values -- title: "Packet Latency: <area, scaling, features, framesize, cores, ...>", - top, centered, font size 18; configurable in specification file: visible / - hidden, text -- x-axis: text, font size 16, bottom; the order of displayed tests is - configurable in the specification file -- x-axis label: "Direction", bottom, centered -- y-axis: integers, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Packet Latency min/avg/max [uSec]", middle, left -- legend: "Indexed Test Cases [idx]", bottom, left, font size 16 - -.. _HTTP-TCP Performance: - -HTTP/TCP Performance -```````````````````` - -The "HTTP/TCP Performance" graph will display the measured data using -statistical box graph separately for "Connections per second" and "Requests per -second". Each data point is constructed from 10 samples. The statistical data -are displayed as hover information. - -.. image:: pic/graph-http-cps.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "HTTP/TCP Performance" not found. - -.. image:: pic/graph-http-rps.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "HTTP/TCP Performance" not found. - -**Description:** - -*Data displayed:* - -- requests / connections per second, the same tests configured for 1, 2 and - 4 cores (3 data points in each graph) -- x-axis: indexed test cases -- y-axis: requests/connections per second, linear scale, beginning with 0 -- hover information: statistical data (min, lower fence, q1, median, q3, - higher fence, max), test case name - -*Layout:* - -- plot type: statistical box (plotly.graph_objs.Box) -- data series format: box -- title: "VPP HTTP Server Performance", top, centered, font size 18 -- x-axis: integers, font size 16, bottom -- x-axis label: "Indices of Test Cases [Index]", bottom, centered, font size 16 -- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Connections per second [cps]", "Requests per second [rps]", - top, left -- legend: "Indexed Test Cases [idx]", bottom, left, font size 16 - -New graphs to be added ----------------------- - -- *Comparison between releases* - - - compare MRR, NDR, PDR between releases - - use as many releases as available - -- *Comparison between processor architectures* - - - compare MRR, NDR, PDR between processor architectures - - HSW vs SKX (vs ARM when available) - -- *Comparison between 2-node and 3-node topologies* - - - compare MRR, NDR, PDR between topologies - - 3n-skx vs 2n-skx - -- *Comparison between different physical testbed instances* - - - compare the results of the same test (MRR, NDR, PDR) run on different - instances of the same testbed, e.g. HSW - - HSW vs HSW, SKX vs SKX - -- *Comparison between NICs* - - - compare the results of the same test (MRR, NDR, PDR) run on different NICs - but on the same instance of a physical testbed. - - x520 vs x710 vs xl710 on HSW - - x710 vs xxv710 on SKX - -- *Other comparisons* - -.. note:: - - - Partially based on the existing tables in the Report - - Only selected TCs - -.. _Comparison between releases: - -Comparison between releases -``````````````````````````` - -This graph will compare the results of the same test from different releases. -One graph can present the data from one or more tests logically grouped. See -`Grouping of tests in graphs`_ for more information. -Each data point is constructed from 10 samples. The statistical data are -displayed as hover information. - -.. image:: pic/graph_cmp_releases.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Comparison between releases" not found. - -**Description:** - -*Data displayed:* - -- data: packet throughput -- x-axis: release -- y-axis: packet throughput [Mpps], linear scale, beginning with 0 -- hover information: statistical data (median, stdev), test case name, release - -*Layout:* - -- plot type: scatter with line -- data series format: line with markers -- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>", - top, centered, font size 18 -- x-axis: strings, font size 16, bottom -- x-axis label: "Release", bototm, centered, font size 16 -- y-axis: floats, starting with 0, dynamic range, linear, bottom, font size 16 -- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16 -- legend: "Test Cases", bottom, left, font size 16 - -.. _Comparison between processor architectures: - -Comparison between processor architectures -`````````````````````````````````````````` - -This graph will compare the results of the same test from the same release run -on the different processor architectures (HSW, SKX, later ARM). -One graph can present the data from one or more tests logically grouped. See -`Grouping of tests in graphs`_ for more information. -Each data point is constructed from 10 samples. The statistical data are -displayed as hover information. - -.. image:: pic/graph_cmp_arch.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Comparison between processor architectures" not found. - -**Description:** - -*Data displayed:* - -- data: packet throughput -- x-axis: processor architecture -- y-axis: throughput [Mpps], linear scale, beginning with 0 -- hover information: statistical data (median, stdev), test case name, processor - architecture - -*Layout:* - -- plot type: scatter with line -- data series format: line with markers -- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>", - top, centered, font size 18 -- x-axis: strings, font size 16, bottom, centered -- x-axis label: "Processor architecture", bottom, centered, font size 16 -- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Packet Throughput [Mpps]", middle, left -- legend: "Test cases", bottom, left, font size 16 - -.. _Comparison between 2-node and 3-node topologies: - -Comparison between 2-node and 3-node topologies -``````````````````````````````````````````````` - -This graph will compare the results of the same test from the same release run -on the same processor architecture but different topologies (3n-skx, 2n-skx). -One graph can present the data from one or more tests logically grouped. See -`Grouping of tests in graphs`_ for more information. -Each data point is constructed from 10 samples. The statistical data are -displayed as hover information. - -.. image:: pic/graph_cmp_topo.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Comparison between 2-node and 3-node topologies" not found. - -**Description:** - -*Data displayed:* - -- data: packet throughput -- x-axis: topology -- y-axis: throughput [Mpps], linear scale, beginning with 0 -- hover information: statistical data (median, stdev), test case name, topology - -*Layout:* - -- plot type: scatter with line -- data series format: line with markers -- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>", - top, centered, font size 18 -- x-axis: strings, font size 16, bottom, centered -- x-axis label: "Topology", bottom, centered, font size 16 -- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16 -- legend: "Test cases", bottom, left, font size 16 - -.. _Comparison between different physical testbed instances: - -Comparison between different physical testbed instances -``````````````````````````````````````````````````````` - -This graph will compare the results of the same test from the same release run -on the same processor architecture, the same topology but different physical -testbed instances. -One graph can present the data from one or more tests logically grouped. See -`Grouping of tests in graphs`_ for more information. -Each data point is constructed from 10 samples. The statistical data are -displayed as hover information. - - -.. image:: pic/graph_cmp_testbed.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Comparison between different physical testbed instances" not - found. - -**Description:** - -*Data displayed:* - -- data: packet throughput -- x-axis: physical testbed instances -- y-axis: throughput [Mpps], linear scale, beginning with 0 -- hover information: statistical data (median, stdev), test case name, physical - testbed instance - -*Layout:* - -- plot type: scatter with line -- data series format: line with markers -- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>", - top, centered, font size 18 -- x-axis: strings, font size 16, bottom, centered -- x-axis label: "Physical Testbed Instance", bottom, centered, font size 16 -- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16 -- legend: "Test cases", bottom, left, font size 16 - -.. _Comparison between NICs: - -Comparison between NICs -``````````````````````` - -This graph will compare the results of the same test from the same release run -on the same processor architecture, the same topology but different NICs. -One graph can present the data from one or more tests logically grouped. See -`Grouping of tests in graphs`_ for more information. -Each data point is constructed from 10 samples. The statistical data are -displayed as hover information. - -.. image:: pic/graph_cmp_nics.svg - :width: 800 px - :scale: 50 % - :align: center - :alt: Graph "Comparison between NICs" not found. - -**Description:** - -*Data displayed:* - -- data: packet throughput -- x-axis: NICs -- y-axis: packet throughput [Mpps], linear scale, beginning with 0 -- hover information: statistical data (median, stdev), test case name, NIC - -*Layout:* - -- plot type: scatter with line -- data series format: line with markers -- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>", - top, centered, font size 18 -- x-axis: strings, font size 16, bottom -- x-axis label: "NIC", bottom, centered, font size 16 -- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left -- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16 -- legend: "Test cases", bottom, left, font size 16 - -.. _Other comparisons: - -Other comparisons -````````````````` - -**Other tests results comparisons** - -- compare packet throughput for vhost vs memif - -**Other views on collected data** - -- per `Vratko Polak email on csit-dev <https://lists.fd.io/g/csit-dev/message/3008>`_. - -.. _Grouping of tests in graphs: - -Grouping of tests in graphs ---------------------------- - -A graph can present results of one or more tests. The tests are grouped -according to the defined criteria. In the ideal case, all graphs use the same -groups of tests. - -The grouping of tests is described in a separate document. - -.. TODO: [MK], [TF]: Create the document. -.. TODO: [TF]: Add the link. -.. TODO: [TF]: Remove/edit the next paragraph when the document is ready. - -**Example of data grouping:** - -- ip4: ip4base, ip4scale20k, ip4scale200k, ip4scale2m - - data presented in this order from left to right -- ip6: similar to ip4 -- l2bd: similar to ip4. - -.. _Sorting of tests presented in a graph: - -Sorting of tests presented in a graph -------------------------------------- - -It is possible to specify the order of tests (suites) on the x-axis presented in -a graph: - -- `Packet Throughput`_ -- `Packet Latency`_ - -It is possible to specify the order of tests (suites) in the legend presented in -a graph: - -- `Speedup Multi-core`_ - -In both cases the order is defined in the specification file for each plot -separately, e.g.: - -.. code:: yaml - - - - type: "plot" - <other parameters> - sort: - - "IP4BASE" - - "FIB_20K" - - "FIB_200K" - - "FIB_2M" - -The sorting is based on tags. If more then one test has the same tag, only the -first one is taken. The remaining tests and the tests without listed tags are -placed at the end of the list in random order. - -.. _export static images: - -Export of static images ------------------------ - -Not implemented yet. For more information see: - -- `Plotly: Static image export <https://plot.ly/python/static-image-export/>`_ -- prefered vector formats (svg, pdf) -- requirements: - - plotly-orca - - `Orca <https://github.com/plotly/orca>`_ - - `Orca releases <https://github.com/plotly/orca/releases>`_ - - `Orca management in Python <https://plot.ly/python/orca-management/>`_ - - psutil - -.. _Physical performance limits: - -Physical performance limits ---------------------------- - -+-----------------+----------------+ -| Ethernet links | pps @64B | -+=================+================+ -| 10ge | 14,880,952.38 | -+-----------------+----------------+ -| 25ge | 37,202,380.95 | -+-----------------+----------------+ -| 40ge | 59,523,809.52 | -+-----------------+----------------+ -| 100ge | 148,809,523.81 | -+-----------------+----------------+ - - -+-----------------+----------------+ -| Ethernet links | bps | -+=================+================+ -| 64B | | -+-----------------+----------------+ -| IMIX | | -+-----------------+----------------+ -| 1518B | | -+-----------------+----------------+ -| 9000B | | -+-----------------+----------------+ - - -+-----------------+----------------+ -| NIC | pps @64B | -+=================+================+ -| x520 | 24,460,000 | -+-----------------+----------------+ -| x710 | 35,800,000 | -+-----------------+----------------+ -| xxv710 | 35,800,000 | -+-----------------+----------------+ -| xl710 | 35,800,000 | -+-----------------+----------------+ - - -+-----------------+----------------+ -| NIC | bw ??B | -+=================+================+ -| x520 | | -+-----------------+----------------+ -| x710 | | -+-----------------+----------------+ -| xxv710 | | -+-----------------+----------------+ -| xl710 | | -+-----------------+----------------+ - - -+-----------------+----------------+ -| PCIe | bps | -+=================+================+ -| PCIe Gen3 x8 | 50,000,000,000 | -+-----------------+----------------+ -| PCIe Gen3 x16 | 100,000,000,000| -+-----------------+----------------+ - - -+-----------------+----------------+ -| PCIe | pps @64B | -+=================+================+ -| PCIe Gen3 x8 | 74,404,761.90 | -+-----------------+----------------+ -| PCIe Gen3 x16 | 148,809,523.81 | -+-----------------+----------------+ |