Since the linked fixes: commit, err is returned uninitialized due to the
removal of "return 0". Initialize err to fix it.
This fixes the following intermittent test failure on release builds:
$ perf test "testsuite_probe"
...
-- [ FAIL ] -- perf_probe :: test_invalid_options :: mutually exclusive options :: -L foo -V bar (output regexp parsing)
Regexp not found: \"Error: switch .+ cannot be used with switch .+\"
...
Fixes: 080e47b2a2 ("perf probe: Introduce quotation marks support")
Tested-by: Namhyung Kim <namhyung@kernel.org>
Reviewed-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Signed-off-by: James Clark <james.clark@linaro.org>
Link: https://lore.kernel.org/r/20241211085525.519458-2-james.clark@linaro.org
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Replace unacceptable characters with '_' when generating event name from
the probing function name.
This is not for a C program. For the a C program, it will continue to
remove suffixes.
Note that this language checking depends on the debuginfo. So without
the debuginfo, perf probe will always replaces unacceptable characters
with '_'.
For example.
$ ./perf probe -x cro3 -D \"cro3::cmd::servo::run_show\"
p:probe_cro3/cro3_cmd_servo_run_show /work/cro3/target/x86_64-unknown-linux-gnu/debug/cro3:0x197530
$ ./perf probe -x /work/go/example/outyet/main -D 'main.(*Server).poll'
p:probe_main/main_Server_poll /work/go/example/outyet/main:0x353040
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Lobakin <aleksander.lobakin@intel.com>
Cc: Dima Kogan <dima@secretsauce.net>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Przemek Kitszel <przemyslaw.kitszel@intel.com>
Link: https://lore.kernel.org/r/173145728160.2747044.18089011235495186810.stgit@mhiramat.roam.corp.google.com
[ Removed some extra tabs in the new struct fields ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In non-C languages, it is possible to have ':' in the function names.
It is possible to escape it with backslashes, but if there are too many
backslashes, it is annoying.
This introduce quotation marks (`"` or `'`) support.
For example, without quotes, we have to pass it as below
$ perf probe -x cro3 -L "cro3\:\:cmd\:\:servo\:\:run_show"
<run_show@/work/cro3/src/cmd/servo.rs:0>
0 fn run_show(args: &ArgsShow) -> Result<()> {
1 let list = ServoList::discover()?;
2 let s = list.find_by_serial(&args.servo)?;
3 if args.json {
4 println!("{s}");
With quotes, we can more naturally write the function name as below;
$ perf probe -x cro3 -L \"cro3::cmd::servo::run_show\"
<run_show@/work/cro3/src/cmd/servo.rs:0>
0 fn run_show(args: &ArgsShow) -> Result<()> {
1 let list = ServoList::discover()?;
2 let s = list.find_by_serial(&args.servo)?;
3 if args.json {
4 println!("{s}");
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Lobakin <aleksander.lobakin@intel.com>
Cc: Dima Kogan <dima@secretsauce.net>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Przemek Kitszel <przemyslaw.kitszel@intel.com>
Link: https://lore.kernel.org/r/173099116941.2431889.11609129616090100386.stgit@mhiramat.roam.corp.google.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
If a symbol name is longer than the maximum event length (64 bytes), the
perf tool reports error:
# perf probe -x test_cpp_mangle --add "this_is_a_very_very_long_print_data_abcdefghijklmnopqrstuvwxyz(int)"
snprintf() failed: -7; the event name nbase='this_is_a_very_very_long_print_data_abcdefghijklmnopqrstuvwxyz(int)' is too long
Error: Failed to add events.
An information is missed in the log that the symbol name and the event
name can be set separately. Especially, this is recommended for adding
probe for a long symbol.
This commit refines the log for reminding event syntax.
After:
# perf probe -x test_cpp_mangle --add "this_is_a_very_very_long_print_data_abcdefghijklmnopqrstuvwxyz(int)"
snprintf() failed: -7; the event name 'this_is_a_very_very_long_print_data_abcdefghijklmnopqrstuvwxyz(int)' is too long
Hint: Set a shorter event with syntax "EVENT=PROBEDEF"
EVENT: Event name (max length: 64 bytes).
Error: Failed to add events.
Signed-off-by: Leo Yan <leo.yan@arm.com>
Acked-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Link: https://lore.kernel.org/r/20241012204725.928794-4-leo.yan@arm.com
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
This prototype is obtained indirectly, by luck, from some other header
in probe-event.c in most systems, but recently exploded on alpine:edge:
8 13.39 alpine:edge : FAIL gcc version 13.2.1 20240309 (Alpine 13.2.1_git20240309)
util/probe-event.c: In function 'convert_exec_to_group':
util/probe-event.c:225:16: error: implicit declaration of function 'basename' [-Werror=implicit-function-declaration]
225 | ptr1 = basename(exec_copy);
| ^~~~~~~~
util/probe-event.c:225:14: error: assignment to 'char *' from 'int' makes pointer from integer without a cast [-Werror=int-conversion]
225 | ptr1 = basename(exec_copy);
| ^
cc1: all warnings being treated as errors
make[3]: *** [/git/perf-6.8.0/tools/build/Makefile.build:158: util] Error 2
Fix it by adding the libgen.h header where basename() is prototyped.
Fixes: fb7345bbf7 ("perf probe: Support basic dwarf-based operations on uprobe events")
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/lkml/
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Building perf with EXTRA_CFLAGS="-fsanitize=address" a leak is detect
when trying to add a probe to a non-existent function:
# perf probe -x ~/bin/perf dso__neW
Probe point 'dso__neW' not found.
Error: Failed to add events.
=================================================================
==296634==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 128 byte(s) in 1 object(s) allocated from:
#0 0x7f67642ba097 in calloc (/lib64/libasan.so.8+0xba097)
#1 0x7f67641a76f1 in allocate_cfi (/lib64/libdw.so.1+0x3f6f1)
Direct leak of 65 byte(s) in 1 object(s) allocated from:
#0 0x7f67642b95b5 in __interceptor_realloc.part.0 (/lib64/libasan.so.8+0xb95b5)
#1 0x6cac75 in strbuf_grow util/strbuf.c:64
#2 0x6ca934 in strbuf_init util/strbuf.c:25
#3 0x9337d2 in synthesize_perf_probe_point util/probe-event.c:2018
#4 0x92be51 in try_to_find_probe_trace_events util/probe-event.c:964
#5 0x93d5c6 in convert_to_probe_trace_events util/probe-event.c:3512
#6 0x93d6d5 in convert_perf_probe_events util/probe-event.c:3529
#7 0x56f37f in perf_add_probe_events /var/home/acme/git/perf-tools-next/tools/perf/builtin-probe.c:354
#8 0x572fbc in __cmd_probe /var/home/acme/git/perf-tools-next/tools/perf/builtin-probe.c:738
#9 0x5730f2 in cmd_probe /var/home/acme/git/perf-tools-next/tools/perf/builtin-probe.c:766
#10 0x635d81 in run_builtin /var/home/acme/git/perf-tools-next/tools/perf/perf.c:323
#11 0x6362c1 in handle_internal_command /var/home/acme/git/perf-tools-next/tools/perf/perf.c:377
#12 0x63667a in run_argv /var/home/acme/git/perf-tools-next/tools/perf/perf.c:421
#13 0x636b8d in main /var/home/acme/git/perf-tools-next/tools/perf/perf.c:537
#14 0x7f676302950f in __libc_start_call_main (/lib64/libc.so.6+0x2950f)
SUMMARY: AddressSanitizer: 193 byte(s) leaked in 2 allocation(s).
#
synthesize_perf_probe_point() returns a "detachec" strbuf, i.e. a
malloc'ed string that needs to be free'd.
An audit will be performed to find other such cases.
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lore.kernel.org/lkml/ZM0l1Oxamr4SVjfY@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since @symbol variable access is not supported by uprobe event, it must be
correctly warn user instead of kernel version update.
Committer testing:
With/without the patch:
[root@quaco ~]# perf probe -x ~/bin/perf -L sigtrap_handler
<sigtrap_handler@/home/acme/git/perf-tools-next/tools/perf/tests/sigtrap.c:0>
0 sigtrap_handler(int signum __maybe_unused, siginfo_t *info, void *ucontext __maybe_unused)
1 {
2 if (!__atomic_fetch_add(&ctx.signal_count, 1, __ATOMIC_RELAXED))
3 ctx.first_siginfo = *info;
4 __atomic_fetch_sub(&ctx.tids_want_signal, syscall(SYS_gettid), __ATOMIC_RELAXED);
5 }
static void *test_thread(void *arg)
{
[root@quaco ~]# perf probe -x ~/bin/perf sigtrap_handler:4 "ctx.signal_count"
Without the patch:
[root@quaco ~]# perf probe -x ~/bin/perf sigtrap_handler:4 "ctx.signal_count"
Failed to write event: Invalid argument
Please upgrade your kernel to at least 3.14 to have access to feature @ctx
Error: Failed to add events.
[root@quaco ~]#
With the patch:
[root@quaco ~]#
Failed to write event: Invalid argument
@ctx accesses a variable by symbol name, but that is not supported for user application probe.
Error: Failed to add events.
[root@quaco ~]#
Reported-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Closes: https://lore.kernel.org/all/ZLWDEjvFjrrEJODp@kernel.org/
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Link: https://lore.kernel.org/r/169055397023.67089.12693645664676964310.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The "address" member of "struct probe_trace_point" uses long data type.
If kernel is 64-bit and perf program is 32-bit, size of "address"
variable is 32 bits.
As a result, upper 32 bits of address read from kernel are truncated, an
error occurs during address comparison in kprobe_warn_out_range().
Before:
# perf probe -a schedule
schedule is out of .text, skip it.
Error: Failed to add events.
Solution:
Change data type of "address" variable to u64 and change corresponding
address printing and value assignment.
After:
# perf.new.new probe -a schedule
Added new event:
probe:schedule (on schedule)
You can now use it in all perf tools, such as:
perf record -e probe:schedule -aR sleep 1
# perf probe -l
probe:schedule (on schedule@kernel/sched/core.c)
# perf record -e probe:schedule -aR sleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.156 MB perf.data (1366 samples) ]
# perf report --stdio
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 1K of event 'probe:schedule'
# Event count (approx.): 1366
#
# Overhead Command Shared Object Symbol
# ........ ............... ................. ............
#
6.22% migration/0 [kernel.kallsyms] [k] schedule
6.22% migration/1 [kernel.kallsyms] [k] schedule
6.22% migration/2 [kernel.kallsyms] [k] schedule
6.22% migration/3 [kernel.kallsyms] [k] schedule
6.15% migration/10 [kernel.kallsyms] [k] schedule
6.15% migration/11 [kernel.kallsyms] [k] schedule
6.15% migration/12 [kernel.kallsyms] [k] schedule
6.15% migration/13 [kernel.kallsyms] [k] schedule
6.15% migration/14 [kernel.kallsyms] [k] schedule
6.15% migration/15 [kernel.kallsyms] [k] schedule
6.15% migration/4 [kernel.kallsyms] [k] schedule
6.15% migration/5 [kernel.kallsyms] [k] schedule
6.15% migration/6 [kernel.kallsyms] [k] schedule
6.15% migration/7 [kernel.kallsyms] [k] schedule
6.15% migration/8 [kernel.kallsyms] [k] schedule
6.15% migration/9 [kernel.kallsyms] [k] schedule
0.22% rcu_sched [kernel.kallsyms] [k] schedule
...
#
# (Cannot load tips.txt file, please install perf!)
#
Signed-off-by: Yang Jihong <yangjihong1@huawei.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jianlin Lv <jianlin.lv@arm.com>
Cc: Jin Yao <yao.jin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Li Huafei <lihuafei1@huawei.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Cc: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
Link: http://lore.kernel.org/lkml/20210715063723.11926-1-yangjihong1@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Now the boot-time tracing supports kprobes events and that must be
written in bootconfig file in the following format.
ftrace.event.kprobes.<EVENT_NAME>.probes = <PROBE-DEF>
'perf probe' already supports --definition (-D) action to show probe
definitions, but the format is for tracefs:
[p|r][:EVENT_NAME] <PROBE-DEF>
This patch adds the --bootconfig option for -D action so that it outputs
the probe definitions in bootconfig format. E.g.
$ perf probe --bootconfig -D "path_lookupat:7 err:s32 s:string"
ftrace.event.kprobes.path_lookupat_L7.probe = 'path_lookupat.isra.0+309 err_s32=%ax:s32 s_string=+0(%r13):string'
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lore.kernel.org/lkml/162282412351.452340.14871995440005640114.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
'perf probe' internally checks the probe target is in the text area in
post-process (after analyzing debuginfo). But it fails if the probe
target is in the "inittext".
This is a good limitation for the online kernel because such functions
have gone after booting. However, for using it for boot-time tracing,
user may want to put a probe on init functions.
This skips the post checking process if the target is offline kenrel so
that user can get the probe definition on the init functions.
Without this patch:
$ perf probe -k ./build-x86_64/vmlinux -D do_mount_root:10
Probe point 'do_mount_root:10' not found.
Error: Failed to add events.
With this patch:
$ perf probe -k ./build-x86_64/vmlinux -D do_mount_root:10
p:probe/do_mount_root_L10 mount_block_root+300
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lore.kernel.org/lkml/162282410293.452340.13347006295826431632.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Report possible permission error including kptr_restrict setting
for map__load() failure. This can happen when non-superuser runs
perf probe.
With this patch, perf probe shows the following message.
$ perf probe vfs_read
Failed to load symbols from /proc/kallsyms
Please ensure you can read the /proc/kallsyms symbol addresses.
If the /proc/sys/kernel/kptr_restrict is '2', you can not read
kernel symbol address even if you are a superuser. Please change
it to '1'. If kptr_restrict is '1', the superuser can read the
symbol addresses.
In that case, please run this command again with sudo.
Error: Failed to add events.
Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Link: http://lore.kernel.org/lkml/162282065877.448336.10047912688119745151.stgit@devnote2
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When run as normal user with default sysctl kernel.kptr_restrict=0
and kernel.perf_event_paranoid=2, perf probe fails with:
$ ./perf probe move_page_tables
Relocated base symbol is not found!
The warning message is not much informative. The reason perf fails
is because /proc/kallsyms is restricted by perf_event_paranoid=2
for normal user and thus perf fails to read relocated address of
the base symbol.
Tweaking kptr_restrict and perf_event_paranoid can change the
behavior of perf probe. Also, running as root or privileged user
works too. Add these details in the warning message.
Plus, kmap->ref_reloc_sym might not be always set even if
host_machine is initialized. Above is the example of the same.
Remove that comment.
Reported-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
Signed-off-by: Ravi Bangoria <ravi.bangoria@linux.ibm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lore.kernel.org/lkml/20210525043744.193297-1-ravi.bangoria@linux.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Perf failed to add a kretprobe event with debuginfo of vmlinux which is
compiled by gcc with -fpatchable-function-entry option enabled. The
same issue with kernel module.
Issue:
# perf probe -v 'kernel_clone%return $retval'
......
Writing event: r:probe/kernel_clone__return _text+599624 $retval
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
# cat /sys/kernel/debug/tracing/error_log
[156.75] trace_kprobe: error: Retprobe address must be an function entry
Command: r:probe/kernel_clone__return _text+599624 $retval
^
# llvm-dwarfdump vmlinux |grep -A 10 -w 0x00df2c2b
0x00df2c2b: DW_TAG_subprogram
DW_AT_external (true)
DW_AT_name ("kernel_clone")
DW_AT_decl_file ("/home/code/linux-next/kernel/fork.c")
DW_AT_decl_line (2423)
DW_AT_decl_column (0x07)
DW_AT_prototyped (true)
DW_AT_type (0x00dcd492 "pid_t")
DW_AT_low_pc (0xffff800010092648)
DW_AT_high_pc (0xffff800010092b9c)
DW_AT_frame_base (DW_OP_call_frame_cfa)
# cat /proc/kallsyms |grep kernel_clone
ffff800010092640 T kernel_clone
# readelf -s vmlinux |grep -i kernel_clone
183173: ffff800010092640 1372 FUNC GLOBAL DEFAULT 2 kernel_clone
# objdump -d vmlinux |grep -A 10 -w \<kernel_clone\>:
ffff800010092640 <kernel_clone>:
ffff800010092640: d503201f nop
ffff800010092644: d503201f nop
ffff800010092648: d503233f paciasp
ffff80001009264c: a9b87bfd stp x29, x30, [sp, #-128]!
ffff800010092650: 910003fd mov x29, sp
ffff800010092654: a90153f3 stp x19, x20, [sp, #16]
The entry address of kernel_clone converted by debuginfo is _text+599624
(0x92648), which is consistent with the value of DW_AT_low_pc attribute.
But the symbolic address of kernel_clone from /proc/kallsyms is
ffff800010092640.
This issue is found on arm64, -fpatchable-function-entry=2 is enabled when
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y;
Just as objdump displayed the assembler contents of kernel_clone,
GCC generate 2 NOPs at the beginning of each function.
kprobe_on_func_entry detects that (_text+599624) is not the entry address
of the function, which leads to the failure of adding kretprobe event.
kprobe_on_func_entry
->_kprobe_addr
->kallsyms_lookup_size_offset
->arch_kprobe_on_func_entry // FALSE
The cause of the issue is that the first instruction in the compile unit
indicated by DW_AT_low_pc does not include NOPs.
This issue exists in all gcc versions that support
-fpatchable-function-entry option.
I have reported it to the GCC community:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98776
Currently arm64 and PA-RISC may enable fpatchable-function-entry option.
The kernel compiled with clang does not have this issue.
FIX:
This GCC issue only cause the registration failure of the kretprobe event
which doesn't need debuginfo. So, stop using debuginfo for retprobe.
map will be used to query the probe function address.
Signed-off-by: Jianlin Lv <Jianlin.Lv@arm.com>
Acked-by: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: clang-built-linux@googlegroups.com
Cc: Frank Ch. Eigler <fche@redhat.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nick Desaulniers <ndesaulniers@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sumanth Korikkar <sumanthk@linux.ibm.com>
Link: http://lore.kernel.org/lkml/20210210062646.2377995-1-Jianlin.Lv@arm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
