1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
|
.. This work is licensed under a Creative Commons Attribution 4.0 International License.
.. http://creativecommons.org/licenses/by/4.0
=====================
FTrace Debugging Tool
=====================
About Ftrace
-------------
Ftrace is an internal tracer designed to find what is going on inside the kernel. It can be used
for debugging or analyzing latencies and performance related issues that take place outside of
user-space. Although ftrace is typically considered the function tracer, it is really a frame
work of several assorted tracing utilities.
One of the most common uses of ftrace is the event tracing.
**Note:**
- For KVMFORNFV, Ftrace is preferred as it is in-built kernel tool
- More stable compared to other debugging tools
Version Features
----------------
+-----------------------------+-----------------------------------------------+
| | |
| **Release** | **Features** |
| | |
+=============================+===============================================+
| | - Ftrace Debugging tool is not implemented in |
| Colorado | Colorado release of KVMFORNFV |
| | |
+-----------------------------+-----------------------------------------------+
| | - Ftrace aids in debugging the KVMFORNFV |
| Danube | 4.4-linux-kernel level issues |
| | - Option to disable if not required |
+-----------------------------+-----------------------------------------------+
Implementation of Ftrace
-------------------------
Ftrace uses the debugfs file system to hold the control files as
well as the files to display output.
When debugfs is configured into the kernel (which selecting any ftrace
option will do) the directory /sys/kernel/debug will be created. To mount
this directory, you can add to your /etc/fstab file:
.. code:: bash
debugfs /sys/kernel/debug debugfs defaults 0 0
Or you can mount it at run time with:
.. code:: bash
mount -t debugfs nodev /sys/kernel/debug
Some configurations for Ftrace are used for other purposes, like finding latency or analyzing the system. For the purpose of debugging, the kernel configuration parameters that should be enabled are:
.. code:: bash
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
The above parameters must be enabled in /boot/config-4.4.0-el7.x86_64 i.e., kernel config file for ftrace to work. If not enabled, change the parameter to ``y`` and run.,
.. code:: bash
On CentOS
grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot
Re-check the parameters after reboot before running ftrace.
Files in Ftrace:
----------------
The below is a list of few major files in Ftrace.
``current_tracer:``
This is used to set or display the current tracer that is configured.
``available_tracers:``
This holds the different types of tracers that have been compiled into the kernel. The tracers listed here can be configured by echoing their name into current_tracer.
``tracing_on:``
This sets or displays whether writing to the tracering buffer is enabled. Echo 0 into this file to disable the tracer or 1 to enable it.
``trace:``
This file holds the output of the trace in a human readable format.
``tracing_cpumask:``
This is a mask that lets the user only trace on specified CPUs. The format is a hex string representing the CPUs.
``events:``
It holds event tracepoints (also known as static tracepoints) that have been compiled into the kernel. It shows what event tracepoints exist and how they are grouped by system.
Avaliable Tracers
-----------------
Here is the list of current tracers that may be configured based on usage.
- function
- function_graph
- irqsoff
- preemptoff
- preemptirqsoff
- wakeup
- wakeup_rt
Brief about a few:
``function:``
Function call tracer to trace all kernel functions.
``function_graph:``
Similar to the function tracer except that the function tracer probes the functions on their entry whereas the function graph tracer traces on both entry and exit of the functions.
``nop:``
This is the "trace nothing" tracer. To remove tracers from tracing simply echo "nop" into current_tracer.
Examples:
.. code:: bash
To list available tracers:
[tracing]# cat available_tracers
function_graph function wakeup wakeup_rt preemptoff irqsoff preemptirqsoff nop
Usage:
[tracing]# echo function > current_tracer
[tracing]# cat current_tracer
function
To view output:
[tracing]# cat trace | head -10
To Stop tracing:
[tracing]# echo 0 > tracing_on
To Start/restart tracing:
[tracing]# echo 1 > tracing_on;
Ftrace in KVMFORNFV
-------------------
Ftrace is part of KVMFORNFV D-Release. KVMFORNFV built 4.4-linux-Kernel will be tested by
executing cyclictest and analyzing the results/latency values (max, min, avg) generated.
Ftrace (or) function tracer is a stable kernel inbuilt debugging tool which tests kernel in real
time and outputs a log as part of the code. These output logs are useful in following ways.
- Kernel Debugging.
- Helps in Kernel code optimization and
- Can be used to better understand the kernel level code flow
Ftrace logs for KVMFORNFV can be found `here`_:
.. _here: http://artifacts.opnfv.org/kvmfornfv.html
Ftrace Usage in KVMFORNFV Kernel Debugging:
-------------------------------------------
Kvmfornfv has two scripts in /ci/envs to provide ftrace tool:
- enable_trace.sh
- disable_trace.sh
Enabling Ftrace in KVMFORNFV
----------------------------
The enable_trace.sh script is triggered by changing ftrace_enable value in test_kvmfornfv.sh
script to 1 (which is zero by default). Change as below to enable Ftrace.
.. code:: bash
ftrace_enable=1
Note:
- Ftrace is enabled before
Details of enable_trace script
------------------------------
- CPU coremask is calculated using getcpumask()
- All the required events are enabled by,
echoing "1" to $TRACEDIR/events/event_name/enable file
Example,
.. code:: bash
$TRACEDIR = /sys/kernel/debug/tracing/
sudo bash -c "echo 1 > $TRACEDIR/events/irq/enable"
sudo bash -c "echo 1 > $TRACEDIR/events/task/enable"
sudo bash -c "echo 1 > $TRACEDIR/events/syscalls/enable"
The set_event file contains all the enabled events list
- Function tracer is selected. May be changed to other avaliable tracers based on requirement
.. code:: bash
sudo bash -c "echo function > $TRACEDIR/current_tracer
- When tracing is turned ON by setting ``tracing_on=1``, the ``trace`` file keeps getting append with the traced data until ``tracing_on=0`` and then ftrace_buffer gets cleared.
.. code:: bash
To Stop/Pause,
echo 0 >tracing_on;
To Start/Restart,
echo 1 >tracing_on;
- Once tracing is diabled, disable_trace.sh script is triggered.
Details of disable_trace Script
-------------------------------
In disable trace script the following are done:
- The trace file is copied and moved to /tmp folder based on timestamp
- The current tracer file is set to ``nop``
- The set_event file is cleared i.e., all the enabled events are disabled
- Kernel Ftrace is disabled/unmounted
Publishing Ftrace logs:
-----------------------
The generated trace log is pushed to `artifacts`_ by kvmfornfv-upload-artifact.sh
script available in releng which will be triggered as a part of kvmfornfv daily job.
The `trigger`_ in the script is.,
.. code:: bash
echo "Uploading artifacts for future debugging needs...."
gsutil cp -r $WORKSPACE/build_output/log-*.tar.gz $GS_LOG_LOCATION > $WORKSPACE/gsutil.log 2>&1
.. _artifacts: https://artifacts.opnfv.org/
.. _trigger: https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/kvmfornfv/kvmfornfv-upload-artifact.sh;h=56fb4f9c18a83c689a916dc6c85f9e3ddf2479b2;hb=HEAD#l53
|
