diff options
Diffstat (limited to 'docs/release/userguide/Ftrace.debugging.tool.userguide.rst')
| -rw-r--r-- | docs/release/userguide/Ftrace.debugging.tool.userguide.rst | 259 |
1 files changed, 259 insertions, 0 deletions
diff --git a/docs/release/userguide/Ftrace.debugging.tool.userguide.rst b/docs/release/userguide/Ftrace.debugging.tool.userguide.rst new file mode 100644 index 000000000..95b7f8fe5 --- /dev/null +++ b/docs/release/userguide/Ftrace.debugging.tool.userguide.rst @@ -0,0 +1,259 @@ +.. This work is licensed under a Creative Commons Attribution 4.0 International License. + +.. http://creativecommons.org/licenses/by/4.0 + +===================== +FTrace Debugging Tool +===================== + +About Ftrace +------------- +Ftrace is an internal tracer designed to find what is going on inside the kernel. It can be used +for debugging or analyzing latencies and performance related issues that take place outside of +user-space. Although ftrace is typically considered the function tracer, it is really a frame +work of several assorted tracing utilities. + + One of the most common uses of ftrace is the event tracing. + +**Note:** +- For KVM4NFV, Ftrace is preferred as it is in-built kernel tool +- More stable compared to other debugging tools + +Version Features +---------------- + ++-----------------------------+-----------------------------------------------+ +| | | +| **Release** | **Features** | +| | | ++=============================+===============================================+ +| | - Ftrace Debugging tool is not implemented in | +| Colorado | Colorado release of KVM4NFV | +| | | ++-----------------------------+-----------------------------------------------+ +| | - Ftrace aids in debugging the KVM4NFV | +| Danube | 4.4-linux-kernel level issues | +| | - Option to disable if not required | ++-----------------------------+-----------------------------------------------+ + + +Implementation of Ftrace +------------------------- +Ftrace uses the debugfs file system to hold the control files as +well as the files to display output. + +When debugfs is configured into the kernel (which selecting any ftrace +option will do) the directory /sys/kernel/debug will be created. To mount +this directory, you can add to your /etc/fstab file: + +.. code:: bash + + debugfs /sys/kernel/debug debugfs defaults 0 0 + +Or you can mount it at run time with: + +.. code:: bash + + mount -t debugfs nodev /sys/kernel/debug + +Some configurations for Ftrace are used for other purposes, like finding latency or analyzing the system. For the purpose of debugging, the kernel configuration parameters that should be enabled are: + +.. code:: bash + + CONFIG_FUNCTION_TRACER=y + CONFIG_FUNCTION_GRAPH_TRACER=y + CONFIG_STACK_TRACER=y + CONFIG_DYNAMIC_FTRACE=y + +The above parameters must be enabled in /boot/config-4.4.0-el7.x86_64 i.e., kernel config file for ftrace to work. If not enabled, change the parameter to ``y`` and run., + +.. code:: bash + + On CentOS + grub2-mkconfig -o /boot/grub2/grub.cfg + sudo reboot + +Re-check the parameters after reboot before running ftrace. + +Files in Ftrace: +---------------- +The below is a list of few major files in Ftrace. + + ``current_tracer:`` + + This is used to set or display the current tracer that is configured. + + ``available_tracers:`` + + This holds the different types of tracers that have been compiled into the kernel. The tracers listed here can be configured by echoing their name into current_tracer. + + ``tracing_on:`` + + This sets or displays whether writing to the tracering buffer is enabled. Echo 0 into this file to disable the tracer or 1 to enable it. + + ``trace:`` + + This file holds the output of the trace in a human readable format. + + ``tracing_cpumask:`` + + This is a mask that lets the user only trace on specified CPUs. The format is a hex string representing the CPUs. + + ``events:`` + + It holds event tracepoints (also known as static tracepoints) that have been compiled into the kernel. It shows what event tracepoints exist and how they are grouped by system. + + +Avaliable Tracers +----------------- + +Here is the list of current tracers that may be configured based on usage. + +- function +- function_graph +- irqsoff +- preemptoff +- preemptirqsoff +- wakeup +- wakeup_rt + +Brief about a few: + + ``function:`` + + Function call tracer to trace all kernel functions. + + ``function_graph:`` + + Similar to the function tracer except that the function tracer probes the functions on their entry whereas the function graph tracer traces on both entry and exit of the functions. + + ``nop:`` + + This is the "trace nothing" tracer. To remove tracers from tracing simply echo "nop" into current_tracer. + +Examples: + +.. code:: bash + + + To list available tracers: + [tracing]# cat available_tracers + function_graph function wakeup wakeup_rt preemptoff irqsoff preemptirqsoff nop + + Usage: + [tracing]# echo function > current_tracer + [tracing]# cat current_tracer + function + + To view output: + [tracing]# cat trace | head -10 + + To Stop tracing: + [tracing]# echo 0 > tracing_on + + To Start/restart tracing: + [tracing]# echo 1 > tracing_on; + + +Ftrace in KVM4NFV +----------------- +Ftrace is part of KVM4NFV D-Release. KVM4NFV built 4.4-linux-Kernel will be tested by +executing cyclictest and analyzing the results/latency values (max, min, avg) generated. +Ftrace (or) function tracer is a stable kernel inbuilt debugging tool which tests real time +kernel and outputs a log as part of the code. These output logs are useful in following ways. + + - Kernel Debugging. + - Helps in Kernel code optimization and + - Can be used to better understand the kernel level code flow + +Ftrace logs for KVM4NFV can be found `here`_: + + +.. _here: http://artifacts.opnfv.org/kvmfornfv.html + +Ftrace Usage in KVM4NFV Kernel Debugging: +----------------------------------------- +Kvm4nfv has two scripts in /ci/envs to provide ftrace tool: + + - enable_trace.sh + - disable_trace.sh + +.. code:: bash + + Found at., + $ cd kvmfornfv/ci/envs + +Enabling Ftrace in KVM4NFV +-------------------------- + +The enable_trace.sh script is triggered by changing ftrace_enable value in test_kvmfornfv.sh +script to 1 (which is zero by default). Change as below to enable Ftrace. + +.. code:: bash + + ftrace_enable=1 + +Note: + +- Ftrace is enabled before + +Details of enable_trace script +------------------------------ + +- CPU coremask is calculated using getcpumask() +- All the required events are enabled by, + echoing "1" to $TRACEDIR/events/event_name/enable file + +Example, + +.. code:: bash + + $TRACEDIR = /sys/kernel/debug/tracing/ + sudo bash -c "echo 1 > $TRACEDIR/events/irq/enable" + sudo bash -c "echo 1 > $TRACEDIR/events/task/enable" + sudo bash -c "echo 1 > $TRACEDIR/events/syscalls/enable" + +The set_event file contains all the enabled events list + +- Function tracer is selected. May be changed to other avaliable tracers based on requirement + +.. code:: bash + + sudo bash -c "echo function > $TRACEDIR/current_tracer + +- When tracing is turned ON by setting ``tracing_on=1``, the ``trace`` file keeps getting append with the traced data until ``tracing_on=0`` and then ftrace_buffer gets cleared. + +.. code:: bash + + To Stop/Pause, + echo 0 >tracing_on; + + To Start/Restart, + echo 1 >tracing_on; + +- Once tracing is diabled, disable_trace.sh script is triggered. + +Details of disable_trace Script +------------------------------- +In disable trace script the following are done: + +- The trace file is copied and moved to /tmp folder based on timestamp +- The current tracer file is set to ``nop`` +- The set_event file is cleared i.e., all the enabled events are disabled +- Kernel Ftrace is disabled/unmounted + + +Publishing Ftrace logs: +----------------------- +The generated trace log is pushed to `artifacts`_ by kvmfornfv-upload-artifact.sh +script available in releng which will be triggered as a part of kvm4nfv daily job. +The `trigger`_ in the script is., + +.. code:: bash + + echo "Uploading artifacts for future debugging needs...." + gsutil cp -r $WORKSPACE/build_output/log-*.tar.gz $GS_LOG_LOCATION > $WORKSPACE/gsutil.log 2>&1 + +.. _artifacts: https://artifacts.opnfv.org/ + +.. _trigger: https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/kvmfornfv/kvmfornfv-upload-artifact.sh;h=56fb4f9c18a83c689a916dc6c85f9e3ddf2479b2;hb=HEAD#l53 |