1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
|
=========================
Tracing Ceph With BlkKin
=========================
Ceph can use Blkin, a library created by Marios Kogias and others,
which enables tracking a specific request from the time it enters
the system at higher levels till it is finally served by RADOS.
In general, Blkin implements the Dapper_ tracing semantics
in order to show the causal relationships between the different
processing phases that an IO request may trigger. The goal is an
end-to-end visualisation of the request's route in the system,
accompanied by information concerning latencies in each processing
phase. Thanks to LTTng this can happen with a minimal overhead and
in realtime. The LTTng traces can then be visualized with Twitter's
Zipkin_.
.. _Dapper: http://static.googleusercontent.com/media/research.google.com/el//pubs/archive/36356.pdf
.. _Zipkin: http://zipkin.io/
Installing Blkin
================
You can install Markos Kogias' upstream Blkin_ by hand.::
cd blkin/
make && make install
or build distribution packages using DistroReadyBlkin_, which also comes with
pkgconfig support. If you choose the latter, then you must generate the
configure and make files first.::
cd blkin
autoreconf -i
.. _Blkin: https://github.com/marioskogias/blkin
.. _DistroReadyBlkin: https://github.com/agshew/blkin
Configuring Ceph with Blkin
===========================
If you built and installed Blkin by hand, rather than building and
installing packages, then set these variables before configuring
Ceph.::
export BLKIN_CFLAGS=-Iblkin/
export BLKIN_LIBS=-lzipkin-cpp
Since there are separate lttng and blkin changes to Ceph, you may
want to configure with something like::
./configure --with-blkin --without-lttng --with-debug
Testing Blkin
=============
It's easy to test Ceph's Blkin tracing. Let's assume you don't have
Ceph already running, and you compiled Ceph with Blkin support but
you did't install it. Then launch Ceph with the ``vstart.sh`` script
in Ceph's src directgory so you can see the possible tracepoints.::
cd src
OSD=3 MON=3 RGW=1 ./vstart.sh -n
lttng list --userspace
You'll see something like the following:::
UST events:
-------------
PID: 8987 - Name: ./ceph-osd
zipkin:timestamp (loglevel: TRACE_WARNING (4)) (type: tracepoint)
zipkin:keyval (loglevel: TRACE_WARNING (4)) (type: tracepoint)
ust_baddr_statedump:soinfo (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
PID: 8407 - Name: ./ceph-mon
zipkin:timestamp (loglevel: TRACE_WARNING (4)) (type: tracepoint)
zipkin:keyval (loglevel: TRACE_WARNING (4)) (type: tracepoint)
ust_baddr_statedump:soinfo (loglevel: TRACE_DEBUG_LINE (13)) (type: tracepoint)
...
Next, stop Ceph so that the tracepoints can be enabled.::
./stop.sh
Start up an LTTng session and enable the tracepoints.::
lttng create blkin-test
lttng enable-event --userspace zipkin:timestamp
lttng enable-event --userspace zipkin:keyval
lttng start
Then start up Ceph again.::
OSD=3 MON=3 RGW=1 ./vstart.sh -n
You may want to check that ceph is up.::
./ceph status
Now put something in usin rados, check that it made it, get it back, and remove it.::
./rados mkpool test-blkin
./rados put test-object-1 ./vstart.sh --pool=test-blkin
./rados -p test-blkin ls
./ceph osd map test-blkin test-object-1
./rados get test-object-1 ./vstart-copy.sh --pool=test-blkin
md5sum vstart*
./rados rm test-object-1 --pool=test-blkin
You could also use the example in ``examples/librados/`` or ``rados bench``.
Then stop the LTTng session and see what was collected.::
lttng stop
lttng view
You'll see something like:::
[13:09:07.755054973] (+?.?????????) scruffy zipkin:timestamp: { cpu_id = 5 }, { trace_name = "Main", service_name = "MOSDOp", port_no = 0, ip = "0.0.0.0", trace_id = 7492589359882233221, span_id = 2694140257089376129, parent_span_id = 0, event = "Message allocated" }
[13:09:07.755071569] (+0.000016596) scruffy zipkin:keyval: { cpu_id = 5 }, { trace_name = "Main", service_name = "MOSDOp", port_no = 0, ip = "0.0.0.0", trace_id = 7492589359882233221, span_id = 2694140257089376129, parent_span_id = 0, key = "Type", val = "MOSDOp" }
[13:09:07.755074217] (+0.000002648) scruffy zipkin:keyval: { cpu_id = 5 }, { trace_name = "Main", service_name = "MOSDOp", port_no = 0, ip = "0.0.0.0", trace_id = 7492589359882233221, span_id = 2694140257089376129, parent_span_id = 0, key = "Reqid", val = "client.4126.0:1" }
...
Install Zipkin
===============
One of the points of using Blkin is so that you can look at the traces
using Zipkin. Users should run Zipkin as a tracepoints collector and
also a web service, which means users need to run three services,
zipkin-collector, zipkin-query and zipkin-web.
Download Zipkin Package::
wget https://github.com/twitter/zipkin/archive/1.1.0.tar.gz
tar zxf 1.1.0.tar.gz
cd zipkin-1.1.0
bin/collector cassandra &
bin/query cassandra &
bin/web &
Check Zipkin::
bin/test
Browse http://${zipkin-web-ip}:8080
Show Ceph's Blkin Traces in Zipkin-web
======================================
Blkin provides a script which translates lttng result to Zipkin
(Dapper) semantics.
Send lttng data to Zipkin::
python3 babeltrace_zipkin.py ${lttng-traces-dir}/${blkin-test}/ust/uid/0/64-bit/ -p ${zipkin-collector-port(9410 by default)} -s ${zipkin-collector-ip}
Example::
python3 babeltrace_zipkin.py ~/lttng-traces-dir/blkin-test-20150225-160222/ust/uid/0/64-bit/ -p 9410 -s 127.0.0.1
Check Ceph traces on webpage::
Browse http://${zipkin-web-ip}:8080
Click "Find traces"
|
