1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
|
Capturing VPP core dumps
========================
In order to debug a crash of VPP, it is required to provide a coredump
file, which allows backtracing of the VPP issue. The following items are
the requirements for capturing a coredump:
1. Disable k8s Probes to Prevent k8s from Restarting the POD with a Crashed VPP
-------------------------------------------------------------------------------
As described in
`BUG_REPORTS.md <BUG_REPORTS.html#collecting-the-logs-in-case-of-crash-loop>`__.
2. Modify VPP Startup config file
---------------------------------
In ``/etc/vpp/contiv-vswitch.conf``, add the following lines into the
``unix`` section:
::
unix {
...
coredump-size unlimited
full-coredump
}
3. Turn on Coredumps in the Vswitch Container
---------------------------------------------
After re-deploying Contiv-VPP networking, enter bash shell in the
vswitch container (use actual name of the vswitch POD -
``contiv-vswitch-7whk7`` in this case):
::
kubectl exec -it contiv-vswitch-7whk7 -n kube-system -c contiv-vswitch bash
Enable coredumps:
::
mkdir -p /tmp/dumps
sysctl -w debug.exception-trace=1
sysctl -w kernel.core_pattern="/tmp/dumps/%e-%t"
ulimit -c unlimited
echo 2 > /proc/sys/fs/suid_dumpable
4. Let VPP Crash
----------------
Now repeat the steps that lead to the VPP crash. You can also force VPP
to crash at the point where it is running (e.g., if it is stuck) by
using the SIGQUIT signal:
::
kill -3 `pidof vpp`
5. Locate and Inspect the Core File
-----------------------------------
The core file should appear in ``/tmp/dumps`` in the container:
::
cd /tmp/dumps
ls
vpp_main-1524124440
You can try to backtrace, after installing gdb:
::
apt-get update && apt-get install gdb
gdb vpp vpp_main-1524124440
(gdb) bt
6. Copy the Core File Out of the Container
------------------------------------------
Finally, copy the core file out of the container. First, while still
inside the container, pack the core file into an archive:
::
cd /tmp/dumps
tar cvzf vppdump.tar.gz vpp_main-1524124440
Now, on the host, determine the docker ID of the container, and then
copy the file out of the host:
::
docker ps | grep vswitch_contiv
d7aceb2e4876 c43a70ac3d01 "/usr/bin/supervisor…" 25 minutes ago Up 25 minutes k8s_contiv-vswitch_contiv-vswitch-zqzn6_kube-system_9923952f-43a6-11e8-be84-080027de08ea_0
docker cp d7aceb2e4876:/tmp/dumps/vppdump.tar.gz .
Now you are ready to file a bug in `jira.fd.io <https://jira.fd.io/>`__
and attach the core file.
|
