diff options
| -rw-r--r-- | docs/cpta/data/index.rst | 8 | ||||
| -rw-r--r-- | docs/cpta/index.rst | 14 | ||||
| -rw-r--r-- | docs/cpta/introduction/index.rst | 17 | ||||
| -rw-r--r-- | docs/cpta/methodology/index.rst | 48 | ||||
| -rw-r--r-- | docs/cpta/trending/index.rst | 4 |
5 files changed, 48 insertions, 43 deletions
diff --git a/docs/cpta/data/index.rst b/docs/cpta/data/index.rst index a81f25e8fb..f9992ed335 100644 --- a/docs/cpta/data/index.rst +++ b/docs/cpta/data/index.rst @@ -1,8 +1,8 @@ -Daily Trending Data -=================== +Performance Data +================ -The data used to generate the trending plots is available in a CSV and -pretty ASCII formats: +Performance test data used to generate the trendline graphs is available +in a CSV and pretty ASCII formats: - `csv format <../_static/vpp/cpta-trending.csv>`_, - `pretty ASCII format <../_static/vpp/cpta-trending.txt>`_. diff --git a/docs/cpta/index.rst b/docs/cpta/index.rst index fb4fc7a808..3fd92c6f79 100644 --- a/docs/cpta/index.rst +++ b/docs/cpta/index.rst @@ -1,11 +1,11 @@ -Continuous Performance Trending and Analysis -============================================ +Continuous Performance Trending +=============================== -This auto-generated document contains VPP performance trending graphs and data. -It is generated using CSIT continuous trending test and analysis jobs and is -updated daily. More detail is available on -`CSIT Performance Trending and Analysis <https://wiki.fd.io/view/CSIT/PerformanceTrendingAnalysis>`_ -wiki page. +This document contains VPP performance dashboard, trendline graphs and +performance test data. It is generated using FD.io CSIT continuous +performance trending and analysis jobs and is updated daily. See +:ref:`trending_methodology` section for more details including trend +analysis and anomaly detection methodology. Contents -------- diff --git a/docs/cpta/introduction/index.rst b/docs/cpta/introduction/index.rst index c724c30bad..016037b067 100644 --- a/docs/cpta/introduction/index.rst +++ b/docs/cpta/introduction/index.rst @@ -8,13 +8,12 @@ Performance dashboard tables provide the latest VPP throughput trend, trend compliance and detected anomalies, all on a per VPP test case basis. Linked trendline graphs enable further drill-down into the trendline compliance, sequence and nature of anomalies, as well as -pointers to performance test builds/logs and VPP builds. - -Performance trending is currently based on the Maximum Receive Rate -(MRR) tests. MRR tests measure the packet forwarding rate under the -maximum load offered by traffic generator over a set trial duration, -regardless of packet loss. See :ref:`trending_methodology` section for -more detail including trend and anomaly calculations. +pointers to performance test builds/logs and VPP builds. Performance +trending is currently based on the Maximum Receive Rate (MRR) tests. MRR +tests measure the packet forwarding rate under the maximum load offered +by traffic generator over a set trial duration, regardless of packet +loss. See :ref:`trending_methodology` section for more detail including +trend and anomaly calculations. Data samples are generated by the CSIT VPP performance trending jobs executed twice a day (target start: every 12 hrs, 02:00, 14:00 UTC). All @@ -36,8 +35,8 @@ Legend to table: Tested VPP worker-thread-core combinations (1t1c, 2t2c, 4t4c) are listed in separate tables in section 1.x. Followed by trending methodology in -section 2. and daily trending graphs in sections 3.x. Daily trending -data used for graphs is provided in sections 4.x. +section 2. and trendline graphs in sections 3.x. Performance test data +used for trendline graphs is provided in sections 4.x. VPP worker on 1t1c ------------------ diff --git a/docs/cpta/methodology/index.rst b/docs/cpta/methodology/index.rst index 5efdfaae32..227dfbcb02 100644 --- a/docs/cpta/methodology/index.rst +++ b/docs/cpta/methodology/index.rst @@ -97,6 +97,8 @@ Outlier Detection Outlier evaluation of test result of value <X> follows the definition from previous section: + :: + Outlier Evaluation Formula Evaluation Result ==================================================== X < (Q1 - 1.5 * IQR) Outlier @@ -109,6 +111,8 @@ To verify compliance of test result of valid value <X> against defined trend metrics and detect anomalies, three simple evaluation formulas are used: + :: + Anomaly Compliance Evaluation Evaluation Formula Confidence Level Result ============================================================================= @@ -129,7 +133,9 @@ ago, TMM[last - 1week] and to the maximum of trend values over last quarter except last week, max(TMM[(last - 3mths)..(last - 1week)]), respectively. This results in following trend compliance calculations: - Trend + :: + + Trend Compliance Metric Change Formula V(alue) R(eference) ============================================================================================= Short-Term Change ((V - R) / R) TMM[last] TMM[last - 1week] @@ -179,15 +185,15 @@ data per test case. PT is designed as follows: #. PT job triggers: - #. Periodic e.g. daily. - #. On-demand gerrit triggered. + - Periodic e.g. daily. + - On-demand gerrit triggered. #. Measurements and data calculations per test case: - #. MRR Max Received Rate + - MRR Max Received Rate - #. Measured: Unlimited tolerance of packet loss. - #. Send packets at link rate, count total received packets, divide + - Measured: Unlimited tolerance of packet loss. + - Send packets at link rate, count total received packets, divide by test trial period. #. Archive MRR per test case. @@ -203,18 +209,18 @@ PA is defined as follows: #. PA job triggers: - #. By PT job at its completion. - #. On-demand gerrit triggered. + - By PT job at its completion. + - On-demand gerrit triggered. #. Download and parse archived historical data and the new data: - #. Download RF output.xml files from latest PT job and compressed + - Download RF output.xml files from latest PT job and compressed archived data. - #. Parse out the data filtering test cases listed in PA specification + - Parse out the data filtering test cases listed in PA specification (part of CSIT PAL specification file). - #. Evalute new data from latest PT job against the rolling window of + - Evalute new data from latest PT job against the rolling window of <N> sets of historical data for trendline calculation, anomaly detection and short-term trend compliance. And against long-term trendline metrics for long-term trend compliance. @@ -222,23 +228,23 @@ PA is defined as follows: #. Calculate trend metrics for the rolling window of <N> sets of historical data: - #. Calculate quartiles Q1, Q2, Q3. - #. Trim outliers using IQR. - #. Calculate TMA and TMSD. - #. Calculate normal trending range per test case based on TMM and + - Calculate quartiles Q1, Q2, Q3. + - Trim outliers using IQR. + - Calculate TMA and TMSD. + - Calculate normal trending range per test case based on TMM and TMSD. #. Evaluate new test data against trend metrics: - #. If within the range of (TMA +/- 3*TMSD) => Result = Pass, + - If within the range of (TMA +/- 3*TMSD) => Result = Pass, Reason = Normal. (to be updated base on the final Jenkins code). - #. If below the range => Result = Fail, Reason = Regression. - #. If above the range => Result = Pass, Reason = Progression. + - If below the range => Result = Fail, Reason = Regression. + - If above the range => Result = Pass, Reason = Progression. #. Generate and publish results - #. Relay evaluation result to job result. (to be updated base on the + - Relay evaluation result to job result. (to be updated base on the final Jenkins code). - #. Generate a new set of trend summary dashboard and graphs. - #. Publish trend dashboard and graphs in html format on + - Generate a new set of trend summary dashboard and graphs. + - Publish trend dashboard and graphs in html format on https://docs.fd.io/. diff --git a/docs/cpta/trending/index.rst b/docs/cpta/trending/index.rst index ed8c63fec3..1560c5ece4 100644 --- a/docs/cpta/trending/index.rst +++ b/docs/cpta/trending/index.rst @@ -1,5 +1,5 @@ -Daily Trending Graphs -===================== +Trendline Graphs +================ .. toctree:: |