aboutsummaryrefslogtreecommitdiffstats
path: root/resources/tools/presentation_new/doc/graphs_improvements.rst
blob: 336faf9748f56e822721715ec38cc428b834d37a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
================================
 Envisioning information by PAL
================================

Introduction
------------

This document describes possible improvements in data presentation provided by
PAL for the `Report <https://docs.fd.io/csit/master/report/>`_ and the
`Trending <https://docs.fd.io/csit/master/trending/>`_

You can generate a standalone html version of this document using e.g.
rst2html5 tool:

.. code:: bash

    rst2html5 --stylesheet graphs_improvements.css graphs_improvements.rst >> graphs_improvements.html

**Modifications of existing graphs**

- `Speedup Multi-core`_
- `Packet Throughput`_
- `Packet Latency`_
- `HTTP-TCP Performance`_

**New graphs to be added**

- `Comparison between releases`_
- `Comparison between processor architectures`_
- `Comparison between 2-node and 3-node topologies`_
- `Comparison between different physical testbed instances`_
- `Comparison between NICs`_
- `Other comparisons`_

**Export of static images**

- low priority
- make possible to `export static images`_ which are available via link on the
  web page.
- vector formats (svg, pdf) are preferred

Priorities
----------

**Target CSIT-18.10**

- `Speedup Multi-core`_
- `Packet Throughput`_

**Nice to have in CSIT-18.10**

.. note::

    Only if above done, and target CSIT-18.10 is in time , otherwise next
    release.

- `Packet Latency`_
- `HTTP-TCP Performance`_

Modifications of existing graphs
--------------------------------

The proposed modifications include the changes in:

- the layout of the graphs,
- the data and way how it is presented,
- the test cases presented in the graphs.

The first two points are described below, the last one will be added later as a
separate chapter.

.. _Speedup Multi-core:

Speedup Multi-core
``````````````````

The "Speedup Multicore" graph will display the measured data together with
perfect values calculated as multiples of the best value measured using one
core. The relative difference between measured and perfect values will be
displayed in the hover next to each data point.

.. image:: pic/graph-speedup.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Speedup Multi-core: not found.

**Description:**

*Data displayed:*

- one or more data series from the same area, keep the number of displayed
  data series as low as possible (max 6)
- x-axis: number of cores
- y-axis: throughput (measured and perfect) [Mpps], linear scale, beginning
  with 0
- hover information: Throughput [Mpps], Speedup [1], Relative difference between
  measured and ideal values [%], Perfect Throughput [%]
- Limits of ethernet links, NICs and PCIe. See `Physical performance limits`_.

*Layout:*

- plot type: lines with data points (plotly.graph_objs.Scatter)
- data series format:
    - measured: solid line with data points
    - perfect: dashed line with data points, the same color as "measured"
- title: "Speedup Multi-core: <area, scaling, features, ...>",
  top, centered, font size 18; configurable in specification file: visible /
  hidden, text
- x-axis: integers, starting with 1 (core), linear, font size 16, bottom
- x-axis label: "Number of cores [qty]", bottom, centered, font size 16
- y-axis: float, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Packet Throughput [Mpps]", middle, left
- legend: list of presented test cases, bottom, left, font size 16; the order
  of displayed tests is configurable in the specification file
- annotation: text: "dashed: perfect<br>solid: measured", top, left,
  font size 16

.. _Packet Throughput:

Packet Throughput
`````````````````

The "Packet Throughput" graph will display the measured data using
statistical box graph. Each data point is constructed from 10 samples.
The statistical data are displayed as hover information.

.. image:: pic/graph-throughput.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Packet Throughput" not found.

**Description:**

*Data displayed:*

- one or more data points from the same area, keep the number of displayed
  data points as low as possible (max 6)
- x-axis: indexed test cases
- y-axis: throughput [Mpps], linear scale, beginning with 0
- hover information: statistical data (min, lower fence, q1, median, q3,
  higher fence, max), test case name

*Layout:*

- plot type: statistical box (plotly.graph_objs.Box)
- data series format: box
- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
  top, centered, font size 18; configurable in specification file: visible /
  hidden, text
- x-axis: integers, starting with 1, linear, font size 16, bottom; the order
  of displayed tests is configurable in the specification file
- x-axis label: "Indices of Test Cases [Index]", bottom, centered, font size 16
- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Packet Throughput [Mpps]", middle, left
- legend: "Indexed Test Cases [idx]", bottom, left, font size 16

.. _Packet Latency:

Packet Latency
``````````````

The "Packet Latency" graph will display the measured data using
statistical box graph. Each data point is constructed from 10 samples.
The statistical data are displayed as hover information.

.. image:: pic/graph-latency.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Packet Latency" not found.

**Description:**

*Data displayed:*

- one or more data points from the same area, keep the number of displayed
  data points as low as possible (max 6)
- x-axis: data flow directions
- y-axis: latency min/avg/max [uSec], linear scale, beginning with 0
- hover information: statistical data (min, avg, max), test case name, direction

*Layout:*

- plot type: scatter with errors (plotly.graph_objs.Scatter)
- data series format: data point with min amd max values
- title: "Packet Latency: <area, scaling, features, framesize, cores, ...>",
  top, centered, font size 18; configurable in specification file: visible /
  hidden, text
- x-axis: text, font size 16, bottom; the order of displayed tests is
  configurable in the specification file
- x-axis label: "Direction", bottom, centered
- y-axis: integers, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Packet Latency min/avg/max [uSec]", middle, left
- legend: "Indexed Test Cases [idx]", bottom, left, font size 16

.. _HTTP-TCP Performance:

HTTP/TCP Performance
````````````````````

The "HTTP/TCP Performance" graph will display the measured data using
statistical box graph separately for "Connections per second" and "Requests per
second". Each data point is constructed from 10 samples. The statistical data
are displayed as hover information.

.. image:: pic/graph-http-cps.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "HTTP/TCP Performance" not found.

.. image:: pic/graph-http-rps.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "HTTP/TCP Performance" not found.

**Description:**

*Data displayed:*

- requests / connections per second, the same tests configured for 1, 2 and
  4 cores (3 data points in each graph)
- x-axis: indexed test cases
- y-axis: requests/connections per second, linear scale, beginning with 0
- hover information: statistical data (min, lower fence, q1, median, q3,
  higher fence, max), test case name

*Layout:*

- plot type: statistical box (plotly.graph_objs.Box)
- data series format: box
- title: "VPP HTTP Server Performance", top, centered, font size 18
- x-axis: integers, font size 16, bottom
- x-axis label: "Indices of Test Cases [Index]", bottom, centered, font size 16
- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Connections per second [cps]", "Requests per second [rps]",
  top, left
- legend: "Indexed Test Cases [idx]", bottom, left, font size 16

New graphs to be added
----------------------

- *Comparison between releases*

    - compare MRR, NDR, PDR between releases
    - use as many releases as available

- *Comparison between processor architectures*

    - compare MRR, NDR, PDR between processor architectures
    - HSW vs SKX (vs ARM when available)

- *Comparison between 2-node and 3-node topologies*

    - compare MRR, NDR, PDR between topologies
    - 3n-skx vs 2n-skx

- *Comparison between different physical testbed instances*

    - compare the results of the same test (MRR, NDR, PDR) run on different
      instances of the same testbed, e.g. HSW
    - HSW vs HSW, SKX vs SKX

- *Comparison between NICs*

    - compare the results of the same test (MRR, NDR, PDR) run on different NICs
      but on the same instance of a physical testbed.
    - x520 vs x710 vs xl710 on HSW
    - x710 vs xxv710 on SKX

- *Other comparisons*

.. note::

    - Partially based on the existing tables in the Report
    - Only selected TCs

.. _Comparison between releases:

Comparison between releases
```````````````````````````

This graph will compare the results of the same test from different releases.
One graph can present the data from one or more tests logically grouped. See
`Grouping of tests in graphs`_ for more information.
Each data point is constructed from 10 samples. The statistical data are
displayed as hover information.

.. image:: pic/graph_cmp_releases.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Comparison between releases" not found.

**Description:**

*Data displayed:*

- data: packet throughput
- x-axis: release
- y-axis: packet throughput [Mpps], linear scale, beginning with 0
- hover information: statistical data (median, stdev), test case name, release

*Layout:*

- plot type: scatter with line
- data series format: line with markers
- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
  top, centered, font size 18
- x-axis: strings, font size 16, bottom
- x-axis label: "Release", bototm, centered, font size 16
- y-axis: floats, starting with 0, dynamic range, linear, bottom, font size 16
- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
- legend: "Test Cases", bottom, left, font size 16

.. _Comparison between processor architectures:

Comparison between processor architectures
``````````````````````````````````````````

This graph will compare the results of the same test from the same release run
on the different processor architectures (HSW, SKX, later ARM).
One graph can present the data from one or more tests logically grouped. See
`Grouping of tests in graphs`_ for more information.
Each data point is constructed from 10 samples. The statistical data are
displayed as hover information.

.. image:: pic/graph_cmp_arch.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Comparison between processor architectures" not found.

**Description:**

*Data displayed:*

- data: packet throughput
- x-axis: processor architecture
- y-axis: throughput [Mpps], linear scale, beginning with 0
- hover information: statistical data (median, stdev), test case name, processor
  architecture

*Layout:*

- plot type: scatter with line
- data series format: line with markers
- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
  top, centered, font size 18
- x-axis: strings, font size 16, bottom, centered
- x-axis label: "Processor architecture", bottom, centered, font size 16
- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Packet Throughput [Mpps]", middle, left
- legend: "Test cases", bottom, left, font size 16

.. _Comparison between 2-node and 3-node topologies:

Comparison between 2-node and 3-node topologies
```````````````````````````````````````````````

This graph will compare the results of the same test from the same release run
on the same processor architecture but different topologies (3n-skx, 2n-skx).
One graph can present the data from one or more tests logically grouped. See
`Grouping of tests in graphs`_ for more information.
Each data point is constructed from 10 samples. The statistical data are
displayed as hover information.

.. image:: pic/graph_cmp_topo.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Comparison between 2-node and 3-node topologies" not found.

**Description:**

*Data displayed:*

- data: packet throughput
- x-axis: topology
- y-axis: throughput [Mpps], linear scale, beginning with 0
- hover information: statistical data (median, stdev), test case name, topology

*Layout:*

- plot type: scatter with line
- data series format: line with markers
- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
  top, centered, font size 18
- x-axis: strings, font size 16, bottom, centered
- x-axis label: "Topology", bottom, centered, font size 16
- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
- legend: "Test cases", bottom, left, font size 16

.. _Comparison between different physical testbed instances:

Comparison between different physical testbed instances
```````````````````````````````````````````````````````

This graph will compare the results of the same test from the same release run
on the same processor architecture, the same topology but different physical
testbed instances.
One graph can present the data from one or more tests logically grouped. See
`Grouping of tests in graphs`_ for more information.
Each data point is constructed from 10 samples. The statistical data are
displayed as hover information.


.. image:: pic/graph_cmp_testbed.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Comparison between different physical testbed instances" not
          found.

**Description:**

*Data displayed:*

- data: packet throughput
- x-axis: physical testbed instances
- y-axis: throughput [Mpps], linear scale, beginning with 0
- hover information: statistical data (median, stdev), test case name, physical
  testbed instance

*Layout:*

- plot type: scatter with line
- data series format: line with markers
- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
  top, centered, font size 18
- x-axis: strings, font size 16, bottom, centered
- x-axis label: "Physical Testbed Instance", bottom, centered, font size 16
- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
- legend: "Test cases", bottom, left, font size 16

.. _Comparison between NICs:

Comparison between NICs
```````````````````````

This graph will compare the results of the same test from the same release run
on the same processor architecture, the same topology but different NICs.
One graph can present the data from one or more tests logically grouped. See
`Grouping of tests in graphs`_ for more information.
Each data point is constructed from 10 samples. The statistical data are
displayed as hover information.

.. image:: pic/graph_cmp_nics.svg
    :width: 800 px
    :scale: 50 %
    :align: center
    :alt: Graph "Comparison between NICs" not found.

**Description:**

*Data displayed:*

- data: packet throughput
- x-axis: NICs
- y-axis: packet throughput [Mpps], linear scale, beginning with 0
- hover information: statistical data (median, stdev), test case name, NIC

*Layout:*

- plot type: scatter with line
- data series format: line with markers
- title: "Packet Throughput: <area, scaling, features, framesize, cores, ...>",
  top, centered, font size 18
- x-axis: strings, font size 16, bottom
- x-axis label: "NIC", bottom, centered, font size 16
- y-axis: floats, starting with 0, dynamic range, linear, font size 16, left
- y-axis label: "Packet Throughput [Mpps]", middle, left, font size 16
- legend: "Test cases", bottom, left, font size 16

.. _Other comparisons:

Other comparisons
`````````````````

**Other tests results comparisons**

- compare packet throughput for vhost vs memif

**Other views on collected data**

- per `Vratko Polak email on csit-dev <https://lists.fd.io/g/csit-dev/message/3008>`_.

.. _Grouping of tests in graphs:

Grouping of tests in graphs
---------------------------

A graph can present results of one or more tests. The tests are grouped
according to the defined criteria. In the ideal case, all graphs use the same
groups of tests.

The grouping of tests is described in a separate document.

.. TODO: [MK], [TF]: Create the document.
.. TODO: [TF]: Add the link.
.. TODO: [TF]: Remove/edit the next paragraph when the document is ready.

**Example of data grouping:**

- ip4: ip4base, ip4scale20k, ip4scale200k, ip4scale2m
    - data presented in this order from left to right
- ip6: similar to ip4
- l2bd: similar to ip4.

.. _Sorting of tests presented in a graph:

Sorting of tests presented in a graph
-------------------------------------

It is possible to specify the order of tests (suites) on the x-axis presented in
a graph:

- `Packet Throughput`_
- `Packet Latency`_

It is possible to specify the order of tests (suites) in the legend presented in
a graph:

- `Speedup Multi-core`_

In both cases the order is defined in the specification file for each plot
separately, e.g.:

.. code:: yaml

    -
      type: "plot"
      <other parameters>
      sort:
      - "IP4BASE"
      - "FIB_20K"
      - "FIB_200K"
      - "FIB_2M"

The sorting is based on tags. If more then one test has the same tag, only the
first one is taken. The remaining tests and the tests without listed tags are
placed at the end of the list in random order.

.. _export static images:

Export of static images
-----------------------

Not implemented yet. For more information see:

- `Plotly: Static image export <https://plot.ly/python/static-image-export/>`_
- prefered vector formats (svg, pdf)
- requirements:
    - plotly-orca
        - `Orca <https://github.com/plotly/orca>`_
        - `Orca releases <https://github.com/plotly/orca/releases>`_
        - `Orca management in Python <https://plot.ly/python/orca-management/>`_
    - psutil

.. _Physical performance limits:

Physical performance limits
---------------------------

+-----------------+----------------+
| Ethernet links  | pps @64B       |
+=================+================+
|  10ge           |  14,880,952.38 |
+-----------------+----------------+
|  25ge           |  37,202,380.95 |
+-----------------+----------------+
|  40ge           |  59,523,809.52 |
+-----------------+----------------+
|  100ge          | 148,809,523.81 |
+-----------------+----------------+


+-----------------+----------------+
| Ethernet links  | bps            |
+=================+================+
| 64B             |                |
+-----------------+----------------+
| IMIX            |                |
+-----------------+----------------+
| 1518B           |                |
+-----------------+----------------+
| 9000B           |                |
+-----------------+----------------+


+-----------------+----------------+
| NIC             | pps @64B       |
+=================+================+
| x520            | 24,460,000     |
+-----------------+----------------+
| x710            | 35,800,000     |
+-----------------+----------------+
| xxv710          | 35,800,000     |
+-----------------+----------------+
| xl710           | 35,800,000     |
+-----------------+----------------+


+-----------------+----------------+
| NIC             | bw ??B         |
+=================+================+
| x520            |                |
+-----------------+----------------+
| x710            |                |
+-----------------+----------------+
| xxv710          |                |
+-----------------+----------------+
| xl710           |                |
+-----------------+----------------+


+-----------------+----------------+
| PCIe            | bps            |
+=================+================+
| PCIe Gen3 x8    | 50,000,000,000 |
+-----------------+----------------+
| PCIe Gen3 x16   | 100,000,000,000|
+-----------------+----------------+


+-----------------+----------------+
| PCIe            | pps @64B       |
+=================+================+
| PCIe Gen3 x8    |  74,404,761.90 |
+-----------------+----------------+
| PCIe Gen3 x16   | 148,809,523.81 |
+-----------------+----------------+