aboutsummaryrefslogtreecommitdiffstats
path: root/docs/cpta/methodology
diff options
context:
space:
mode:
authorVratko Polak <vrpolak@cisco.com>2022-01-14 14:26:53 +0100
committerTibor Frank <tifrank@cisco.com>2022-01-18 09:20:15 +0000
commitb89b8ec88d006a499c0523134244fc8cb47514c8 (patch)
tree6827306ceed6a82a1a451309c3ea588506ed44ca /docs/cpta/methodology
parente9dffba58304835f45e882efe77ea947494f7240 (diff)
Doc: Update anomaly methodology documentation
+ Define currently used compliance metrics (e.g. no short term change), + Clarify that (and why) column names in Dashboard and Alerting are somewhat different from the ones in the methodology document. Change-Id: Ie091b1981d7578245ca3e700820fab9ac93bbd17 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
Diffstat (limited to 'docs/cpta/methodology')
-rw-r--r--docs/cpta/methodology/trend_analysis.rst52
1 files changed, 32 insertions, 20 deletions
diff --git a/docs/cpta/methodology/trend_analysis.rst b/docs/cpta/methodology/trend_analysis.rst
index b98dbe3d37..5b9ebd352d 100644
--- a/docs/cpta/methodology/trend_analysis.rst
+++ b/docs/cpta/methodology/trend_analysis.rst
@@ -16,35 +16,47 @@ Trend Compliance
.. _Trend_Compliance:
-In the text below, "trend at time <t>", shorthand "Trend[t]"
+Trend compliance metrics are targeted to provide an indication of trend
+changes, and hint at their reliability.
+
+There is a difference between compliance metric names used in this document,
+and column names used in :ref:`Dashboard` tables and Alerting emails.
+In cases of low user confusion risk, column names are shortened,
+e.g. Trend instead of Last Trend.
+In cases of high user confusion risk, column names are prolonged,
+e.g. Long-Term Change instead of Trend Change.
+(This document refers to a generic "trend",
+so the compliance metric name is prolonged to Last Trend to avoid confusion.)
+
+The definition of Reference for Trend Change is perhaps surprising.
+It was chosen to allow both positive difference on progression
+(if within last week), but also negative difference on progression
+(if performance was even better somewhere between 3 months and 1 week ago).
+
+In the table below, "trend at time <t>", shorthand "trend[t]"
means "the group average of the group the sample at time <t> belongs to".
Here, time is usually given as "last" or last with an offset,
e.g. "last - 1week".
+Also, "runs[t]" is a shorthand for "number of samples in the group
+the sample at time <t> belongs to".
-Trend compliance metrics are targeted to provide an indication of trend
-changes over a short-term (i.e. weekly) and a long-term (i.e.
-quarterly), comparing the last group average Trend[last], to the one from week
-ago, Trend[last - 1week] and to the maximum of trend values over last
-quarter except last week, max(Trend[last - 3mths]..Trend[last - 1week]),
-respectively.
-
-This results in following trend compliance calculations:
-
-+-------------------------+---------------------------------+-------------+-----------------------------------------------+
-| Trend Compliance Metric | Trend Change Formula | Value | Reference |
-+=========================+=================================+=============+===============================================+
-| Short-Term Change | (Value - Reference) / Reference | Trend[last] | Trend[last - 1week] |
-+-------------------------+---------------------------------+-------------+-----------------------------------------------+
-| Long-Term Change | (Value - Reference) / Reference | Trend[last] | max(Trend[last - 3mths]..Trend[last - 1week]) |
-+-------------------------+---------------------------------+-------------+-----------------------------------------------+
+The definitions of compliance metrics:
-These metrics are displayed in the Dashboard table.
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
+| Compliance Metric | Legend Short Name | Formula | Value | Reference |
++===================+===================+=================================+=============+===============================================+
+| Last Trend | Trend | trend[last] | | |
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
+| Number of runs | Runs | runs[last] | | |
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
+| Trend Change | Long-Term Change | (Value - Reference) / Reference | trend[last] | max(trend[last - 3mths]..trend[last - 1week]) |
++-------------------+-------------------+---------------------------------+-------------+-----------------------------------------------+
Caveats
-------
-Obviously, is result history is too short, the true Trend[t] value
-may not by available, we use the earliest Trend available instead.
+Obviously, if the result history is too short, the true Trend[t] value
+may not by available. We use the earliest Trend available instead.
The current implementaton does not track time of the samples,
it counts runs instead.