1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
|
.. _stats_doc:
Statistics
==========
In VPP most things are measured and counted. There are counters for
interface statistics, like RX, TX counters, packet drops, and so on.
Every node has a set of per-node counters, one set of error counters,
like TTL exceeded, or packet to big or out-of-buffers. And a set of
performance counters, like number of clocks, vectors, calls and
suspends.
There is also a set of system counters and performance counters,
e.g. memory utilization per heap, buffer utilisation and so on.
VPP Counter Architecture
------------------------
Counters are exposed directly via shared memory. These are the actual
counters in VPP, no sampling or aggregation is done by the statistics
infrastructure. With the exception of per node performance data under
/sys/node and a few system counters.
Clients mount the shared memory segment read-only, using a optimistic
concurrency algorithm.
Directory structure as an index.
Memory layout
~~~~~~~~~~~~~
The memory segment consists of a shared header, containing atomics for
the optimistic concurrency mechanism, and offsets into memory for the
directory vectors. The only data structure used is the VPP vectors. All
pointers are converted to offsets so that client applications can map
the shared memory wherever it pleases.
Directory layout
~~~~~~~~~~~~~~~~
Optimistic concurrency
~~~~~~~~~~~~~~~~~~~~~~
::
/*
* Shared header first in the shared memory segment.
*/
typedef struct {
atomic_int_fast64_t epoch;
atomic_int_fast64_t in_progress;
atomic_int_fast64_t directory_offset;
atomic_int_fast64_t error_offset;
atomic_int_fast64_t stats_offset;
} stat_segment_shared_header_t;
Writer
^^^^^^
On the VPP side there is a single writer (controlled by a spinlock).
When the writer starts it sets in_progress=1, continues with the update
of the data-structures, and when done, bumps epoch++ and sets
in_progress=0.
Readers
^^^^^^^
If in_progress=1, there is no point continuing, so reader sits spinning
on the in_progress flag until it is 0. Then it sets start_epoch = epoch
and continues copying out the counter data it is interested in, while
doing strict boundary checks on all offsets / pointers. When the reader
is done, it checks if in_progress=1 or if epoch != start_epoch. If
either of those are true is discards the data read.
How are counters exposed out of VPP?
------------------------------------
Types of Counters
-----------------
All counters under /err and /if are the directly exposed VPP counters.
- Gauges
- u64 / float
- Interface Counters
- Simple counters, counter_t array of threads of an array of interfaces
- Combined counters, vlib_counter_t array of threads of an array of
interfaces.
Client libraries
----------------
Writing a new client library
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A new client library can either wrap the C library (libvppapiclient.so)
or it can integrate directly with the shared memory. That involves
exchanging a file descriptor over the VPP stats Unix domain socket, and
opening the memory mapped segment.
Python
~~~~~~
::
#!/usr/bin/env python3
from vpp_papi.vpp_stats import VPPStats
stats = VPPStats('/run/vpp/stats.sock')
dir = stats.ls(['^/if', '/err/ip4-input', '/sys/node/ip4-input'])
counters = stats.dump(dir)
# Print the RX counters for the first interface on the first worker core
print ('RX interface core 0, sw_if_index 0', counters['/if/rx'][0][0])
C
~
::
#include <vpp-api/client/stat_client.h>
#include <vppinfra/vec.h>
int main (int argc, char **argv) {
uint8_t *patterns = 0;
vec_add1(patterns, "^/if");
vec_add1(patterns, "ip4-input");
int rv = stat_segment_connect(STAT_SEGMENT_SOCKET_FILE);
uint32_t *dir = stat_segment_ls(patterns);
stat_segment_data_t *res = stat_segment_dump(dir);
for (int i = 0; i < vec_len(res); i++) {
switch (res[i].type) {
case STAT_DIR_TYPE_COUNTER_VECTOR_SIMPLE:
for (k = 0; k < vec_len (res[i].simple_counter_vec) - 1; k++)
for (j = 0; j < vec_len (res[i].simple_counter_vec[k]); j++)
fformat (stdout, "[%d @ %d]: %llu packets %s\n",
j, k, res[i].simple_counter_vec[k][j],
res[i].name);
break;
case STAT_DIR_TYPE_COUNTER_VECTOR_COMBINED:
for (k = 0; k < vec_len (res[i].combined_counter_vec); k++)
for (j = 0; j < vec_len (res[i].combined_counter_vec[k]); j++)
fformat (stdout, "[%d @ %d]: %llu packets, %llu bytes %s\n",
j, k, res[i].combined_counter_vec[k][j].packets,
res[i].combined_counter_vec[k][j].bytes,
res[i].name);
break;
case STAT_DIR_TYPE_ERROR_INDEX:
for (j = 0; j < vec_len (res[i].error_vector); j++)
fformat (stdout, "[@%d] %llu %s\n", j, res[i].error_vector[j], res[i].name);
break;
case STAT_DIR_TYPE_SCALAR_INDEX:
fformat (stdout, "%.2f %s\n", res[i].scalar_value, res[i].name);
break;
default:
;
}
}
stat_segment_data_free (res);
}
Integrations
------------
- CLI command. vpp_get_stats [ls \| dump \| poll]
- Prometheus
Future evolution
----------------
- Deprecate the stats over binary API calls that are based on
want_stats
|
