Skip to content

Commit 8823851

Browse files
rostedtrostedt
committed
tracing/histogram: Document variable stacktrace
Add a little documentation (and a useful example) of how a stacktrace can be used within a histogram variable and synthetic event. Link: https://lkml.kernel.org/r/20230117152236.320181354@goodmis.org Cc: Masami Hiramatsu <mhiramat@kernel.org> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Tom Zanussi <zanussi@kernel.org> Cc: Ross Zwisler <zwisler@google.com> Cc: Ching-lin Yu <chinglinyu@google.com> Signed-off-by: Steven Rostedt (Google) <rostedt@goodmis.org>
1 parent cc5fc8b commit 8823851

1 file changed

Lines changed: 129 additions & 0 deletions

File tree

‎Documentation/trace/histogram.rst‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ Documentation written by Tom Zanussi
8181
.usecs display a common_timestamp in microseconds
8282
.percent display a number of percentage value
8383
.graph display a bar-graph of a value
84+
.stacktrace display as a stacktrace (must by a long[] type)
8485
============= =================================================
8586

8687
Note that in general the semantics of a given field aren't
@@ -1786,6 +1787,8 @@ or assigned to a variable and referenced in a subsequent expression::
17861787
# echo 'hist:keys=next_pid:us_per_sec=1000000 ...' >> event/trigger
17871788
# echo 'hist:keys=next_pid:timestamp_secs=common_timestamp/$us_per_sec ...' >> event/trigger
17881789

1790+
Variables can even hold stacktraces, which are useful with synthetic events.
1791+
17891792
2.2.2 Synthetic Events
17901793
----------------------
17911794

@@ -1940,6 +1943,132 @@ the ".buckets" modifier and specify a size (in this case groups of 10).
19401943
Entries: 16
19411944
Dropped: 0
19421945

1946+
To save stacktraces, create a synthetic event with a field of type "unsigned long[]"
1947+
or even just "long[]". For example, to see how long a task is blocked in an
1948+
uninterruptible state:
1949+
1950+
# cd /sys/kernel/tracing
1951+
# echo 's:block_lat pid_t pid; u64 delta; unsigned long[] stack;' > dynamic_events
1952+
# echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace if prev_state == 2' >> events/sched/sched_switch/trigger
1953+
# echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,s=$st:onmax($delta).trace(block_lat,prev_pid,$delta,$s)' >> events/sched/sched_switch/trigger
1954+
# echo 1 > events/synthetic/block_lat/enable
1955+
# cat trace
1956+
1957+
# tracer: nop
1958+
#
1959+
# entries-in-buffer/entries-written: 2/2 #P:8
1960+
#
1961+
# _-----=> irqs-off/BH-disabled
1962+
# / _----=> need-resched
1963+
# | / _---=> hardirq/softirq
1964+
# || / _--=> preempt-depth
1965+
# ||| / _-=> migrate-disable
1966+
# |||| / delay
1967+
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
1968+
# | | | ||||| | |
1969+
<idle>-0 [005] d..4. 521.164922: block_lat: pid=0 delta=8322 stack=STACK:
1970+
=> __schedule+0x448/0x7b0
1971+
=> schedule+0x5a/0xb0
1972+
=> io_schedule+0x42/0x70
1973+
=> bit_wait_io+0xd/0x60
1974+
=> __wait_on_bit+0x4b/0x140
1975+
=> out_of_line_wait_on_bit+0x91/0xb0
1976+
=> jbd2_journal_commit_transaction+0x1679/0x1a70
1977+
=> kjournald2+0xa9/0x280
1978+
=> kthread+0xe9/0x110
1979+
=> ret_from_fork+0x2c/0x50
1980+
1981+
<...>-2 [004] d..4. 525.184257: block_lat: pid=2 delta=76 stack=STACK:
1982+
=> __schedule+0x448/0x7b0
1983+
=> schedule+0x5a/0xb0
1984+
=> schedule_timeout+0x11a/0x150
1985+
=> wait_for_completion_killable+0x144/0x1f0
1986+
=> __kthread_create_on_node+0xe7/0x1e0
1987+
=> kthread_create_on_node+0x51/0x70
1988+
=> create_worker+0xcc/0x1a0
1989+
=> worker_thread+0x2ad/0x380
1990+
=> kthread+0xe9/0x110
1991+
=> ret_from_fork+0x2c/0x50
1992+
1993+
A synthetic event that has a stacktrace field may use it as a key in histogram:
1994+
1995+
# echo 'hist:delta.buckets=100,stack.stacktrace:sort=delta' > events/synthetic/block_lat/trigger
1996+
# cat events/synthetic/block_lat/hist
1997+
1998+
# event histogram
1999+
#
2000+
# trigger info: hist:keys=delta.buckets=100,stacktrace:vals=hitcount:sort=delta.buckets=100:size=2048 [active]
2001+
#
2002+
2003+
{ delta: ~ 0-99, stacktrace:
2004+
event_hist_trigger+0x464/0x480
2005+
event_triggers_call+0x52/0xe0
2006+
trace_event_buffer_commit+0x193/0x250
2007+
trace_event_raw_event_sched_switch+0xfc/0x150
2008+
__traceiter_sched_switch+0x41/0x60
2009+
__schedule+0x448/0x7b0
2010+
schedule_idle+0x26/0x40
2011+
cpu_startup_entry+0x19/0x20
2012+
start_secondary+0xed/0xf0
2013+
secondary_startup_64_no_verify+0xe0/0xeb
2014+
} hitcount: 6
2015+
{ delta: ~ 0-99, stacktrace:
2016+
event_hist_trigger+0x464/0x480
2017+
event_triggers_call+0x52/0xe0
2018+
trace_event_buffer_commit+0x193/0x250
2019+
trace_event_raw_event_sched_switch+0xfc/0x150
2020+
__traceiter_sched_switch+0x41/0x60
2021+
__schedule+0x448/0x7b0
2022+
schedule_idle+0x26/0x40
2023+
cpu_startup_entry+0x19/0x20
2024+
__pfx_kernel_init+0x0/0x10
2025+
arch_call_rest_init+0xa/0x24
2026+
start_kernel+0x964/0x98d
2027+
secondary_startup_64_no_verify+0xe0/0xeb
2028+
} hitcount: 3
2029+
{ delta: ~ 0-99, stacktrace:
2030+
event_hist_trigger+0x464/0x480
2031+
event_triggers_call+0x52/0xe0
2032+
trace_event_buffer_commit+0x193/0x250
2033+
trace_event_raw_event_sched_switch+0xfc/0x150
2034+
__traceiter_sched_switch+0x41/0x60
2035+
__schedule+0x448/0x7b0
2036+
schedule+0x5a/0xb0
2037+
worker_thread+0xaf/0x380
2038+
kthread+0xe9/0x110
2039+
ret_from_fork+0x2c/0x50
2040+
} hitcount: 1
2041+
{ delta: ~ 100-199, stacktrace:
2042+
event_hist_trigger+0x464/0x480
2043+
event_triggers_call+0x52/0xe0
2044+
trace_event_buffer_commit+0x193/0x250
2045+
trace_event_raw_event_sched_switch+0xfc/0x150
2046+
__traceiter_sched_switch+0x41/0x60
2047+
__schedule+0x448/0x7b0
2048+
schedule_idle+0x26/0x40
2049+
cpu_startup_entry+0x19/0x20
2050+
start_secondary+0xed/0xf0
2051+
secondary_startup_64_no_verify+0xe0/0xeb
2052+
} hitcount: 15
2053+
[..]
2054+
{ delta: ~ 8500-8599, stacktrace:
2055+
event_hist_trigger+0x464/0x480
2056+
event_triggers_call+0x52/0xe0
2057+
trace_event_buffer_commit+0x193/0x250
2058+
trace_event_raw_event_sched_switch+0xfc/0x150
2059+
__traceiter_sched_switch+0x41/0x60
2060+
__schedule+0x448/0x7b0
2061+
schedule_idle+0x26/0x40
2062+
cpu_startup_entry+0x19/0x20
2063+
start_secondary+0xed/0xf0
2064+
secondary_startup_64_no_verify+0xe0/0xeb
2065+
} hitcount: 1
2066+
2067+
Totals:
2068+
Hits: 89
2069+
Entries: 11
2070+
Dropped: 0
2071+
19432072
2.2.3 Hist trigger 'handlers' and 'actions'
19442073
-------------------------------------------
19452074

0 commit comments

Comments
 (0)