user/sven/linux.git/tools/perf/examples, branch v4.20.5

perf augmented_syscalls: Start collecting pathnames in the BPF program

2018-11-05T15:41:10Z

This is the start of having the raw_syscalls:sys_enter BPF handler collecting pointer arguments, namely pathnames, and with two syscalls that have that pointer in different arguments, "open" as it as its first argument, "openat" as the second. With this in place the existing beautifiers in 'perf trace' works, those args are shown instead of just the pointer that comes with the syscalls tracepoints. This also serves to show and document pitfalls in the process of using just that place in the kernel (raw_syscalls:sys_enter) plus tables provided by userspace to collect syscall pointer arguments. One is the need to use a barrier, as suggested by Edward, to avoid clang optimizations that make the kernel BPF verifier to refuse loading our pointer contents collector. The end result should be a generic eBPF program that works in all architectures, with the differences amongst archs resolved by the userspace component, 'perf trace', that should get all its tables created automatically from the kernel components where they are defined, via string table constructors for things not expressed in BTF/DWARF (enums, structs, etc), and otherwise using those observability files (BTF). Cc: Adrian Hunter Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: David Ahern Cc: Edward Cree Cc: Jiri Olsa Cc: Martin KaFai Lau Cc: Namhyung Kim Cc: Wang Nan Cc: Yonghong Song Link: https://lkml.kernel.org/n/tip-37dz54pmotgpnwg9tb6zuk9j@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf examples bpf: Start augmenting raw_syscalls:sys_{start,exit}

2018-11-01T17:11:45Z

The previous approach of attaching to each syscall showed how it is possible to augment tracepoints and use that augmentation, pointer payloads, in the existing beautifiers in 'perf trace', but for a more general solution we now will try to augment the main raw_syscalls:sys_{enter,exit} syscalls, and then pass instructions in maps so that it knows which syscalls and which pointer contents, and how many bytes for each of the arguments should be copied. Start with just the bare minimum to collect what is provided by those two tracepoints via the __augmented_syscalls__ map + bpf-output perf event, which results in perf trace showing them without connecting enter+exit: # perf trace -e tools/perf/examples/bpf/augmented_raw_syscalls.c sleep 1 0.000 sleep/11563 raw_syscalls:sys_exit:NR 59 = 0 0.019 ( ): sleep/11563 brk() ... 0.021 sleep/11563 raw_syscalls:sys_exit:NR 12 = 94682642325504 0.033 ( ): sleep/11563 access(filename:, mode: R) ... 0.037 sleep/11563 raw_syscalls:sys_exit:NR 21 = -2 0.041 ( ): sleep/11563 openat(dfd: CWD, filename: , flags: CLOEXEC) ... 0.044 sleep/11563 raw_syscalls:sys_exit:NR 257 = 3 0.045 ( ): sleep/11563 fstat(fd: 3, statbuf: 0x7ffdbf7119b0) ... 0.046 sleep/11563 raw_syscalls:sys_exit:NR 5 = 0 0.047 ( ): sleep/11563 mmap(len: 103334, prot: READ, flags: PRIVATE, fd: 3) ... 0.049 sleep/11563 raw_syscalls:sys_exit:NR 9 = 140196285493248 0.050 ( ): sleep/11563 close(fd: 3) ... 0.051 sleep/11563 raw_syscalls:sys_exit:NR 3 = 0 0.059 ( ): sleep/11563 openat(dfd: CWD, filename: , flags: CLOEXEC) ... 0.062 sleep/11563 raw_syscalls:sys_exit:NR 257 = 3 0.063 ( ): sleep/11563 read(fd: 3, buf: 0x7ffdbf711b78, count: 832) ... 0.065 sleep/11563 raw_syscalls:sys_exit:NR 0 = 832 0.066 ( ): sleep/11563 fstat(fd: 3, statbuf: 0x7ffdbf711a10) ... 0.067 sleep/11563 raw_syscalls:sys_exit:NR 5 = 0 0.068 ( ): sleep/11563 mmap(len: 8192, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS) ... 0.070 sleep/11563 raw_syscalls:sys_exit:NR 9 = 140196285485056 0.073 ( ): sleep/11563 mmap(len: 3889792, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3) ... 0.076 sleep/11563 raw_syscalls:sys_exit:NR 9 = 140196279463936 0.077 ( ): sleep/11563 mprotect(start: 0x7f81fd8a8000, len: 2093056) ... 0.083 sleep/11563 raw_syscalls:sys_exit:NR 10 = 0 0.084 ( ): sleep/11563 mmap(addr: 0x7f81fdaa7000, len: 24576, prot: READ|WRITE, flags: PRIVATE|FIXED|DENYWRITE, fd: 3, off: 1753088) ... 0.088 sleep/11563 raw_syscalls:sys_exit:NR 9 = 140196283314176 0.091 ( ): sleep/11563 mmap(addr: 0x7f81fdaad000, len: 14976, prot: READ|WRITE, flags: PRIVATE|FIXED|ANONYMOUS) ... 0.093 sleep/11563 raw_syscalls:sys_exit:NR 9 = 140196283338752 0.097 ( ): sleep/11563 close(fd: 3) ... 0.098 sleep/11563 raw_syscalls:sys_exit:NR 3 = 0 0.107 ( ): sleep/11563 arch_prctl(option: 4098, arg2: 140196285490432) ... 0.108 sleep/11563 raw_syscalls:sys_exit:NR 158 = 0 0.143 ( ): sleep/11563 mprotect(start: 0x7f81fdaa7000, len: 16384, prot: READ) ... 0.146 sleep/11563 raw_syscalls:sys_exit:NR 10 = 0 0.157 ( ): sleep/11563 mprotect(start: 0x561d037e7000, len: 4096, prot: READ) ... 0.160 sleep/11563 raw_syscalls:sys_exit:NR 10 = 0 0.163 ( ): sleep/11563 mprotect(start: 0x7f81fdcd5000, len: 4096, prot: READ) ... 0.165 sleep/11563 raw_syscalls:sys_exit:NR 10 = 0 0.166 ( ): sleep/11563 munmap(addr: 0x7f81fdcbb000, len: 103334) ... 0.174 sleep/11563 raw_syscalls:sys_exit:NR 11 = 0 0.216 ( ): sleep/11563 brk() ... 0.217 sleep/11563 raw_syscalls:sys_exit:NR 12 = 94682642325504 0.217 ( ): sleep/11563 brk(brk: 0x561d05453000) ... 0.219 sleep/11563 raw_syscalls:sys_exit:NR 12 = 94682642460672 0.220 ( ): sleep/11563 brk() ... 0.221 sleep/11563 raw_syscalls:sys_exit:NR 12 = 94682642460672 0.224 ( ): sleep/11563 open(filename: , flags: CLOEXEC) ... 0.228 sleep/11563 raw_syscalls:sys_exit:NR 2 = 3 0.229 ( ): sleep/11563 fstat(fd: 3, statbuf: 0x7f81fdaacaa0) ... 0.230 sleep/11563 raw_syscalls:sys_exit:NR 5 = 0 0.231 ( ): sleep/11563 mmap(len: 113045344, prot: READ, flags: PRIVATE, fd: 3) ... 0.234 sleep/11563 raw_syscalls:sys_exit:NR 9 = 140196166418432 0.237 ( ): sleep/11563 close(fd: 3) ... 0.238 sleep/11563 raw_syscalls:sys_exit:NR 3 = 0 0.262 ( ): sleep/11563 nanosleep(rqtp: 0x7ffdbf7126f0) ... 1000.399 sleep/11563 raw_syscalls:sys_exit:NR 35 = 0 1000.440 ( ): sleep/11563 close(fd: 1) ... 1000.447 sleep/11563 raw_syscalls:sys_exit:NR 3 = 0 1000.454 ( ): sleep/11563 close(fd: 2) ... 1000.468 ( ): sleep/11563 exit_group( ) # In the next csets we'll connect those events to the existing enter/exit raw_syscalls handlers in 'perf trace', just like we did with the syscalls:sys_{enter,exit}_* tracepoints. Cc: Adrian Hunter Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-5nl8l4hx1tl9pqdx65nkp6pw@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf augmented_syscalls: Avoid optimization to pass older BPF validators

2018-09-03T18:29:53Z

See https://www.spinics.net/lists/netdev/msg480099.html for the whole discussio, but to make the augmented_syscalls.c BPF program to get built and loaded successfully in a greater range of kernels, add an extra check. Related patch: a60dd35d2e39 ("bpf: change bpf_perf_event_output arg5 type to ARG_CONST_SIZE_OR_ZERO") That is in the kernel since v4.15, I couldn't figure why this is hitting me with 4.17.17, but adding the workaround discussed there makes this work with this fedora kernel and with 4.18.recent. Before: # uname -a Linux seventh 4.17.17-100.fc27.x86_64 #1 SMP Mon Aug 20 15:53:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # perf trace -e tools/perf/examples/bpf/augmented_syscalls.c cat /etc/passwd > /dev/null libbpf: load bpf program failed: Permission denied libbpf: -- BEGIN DUMP LOG --- libbpf: 0: (bf) r6 = r1 1: (b7) r1 = 0 2: (7b) *(u64 *)(r10 -8) = r1 3: (7b) *(u64 *)(r10 -16) = r1 4: (7b) *(u64 *)(r10 -24) = r1 5: (7b) *(u64 *)(r10 -32) = r1 6: (7b) *(u64 *)(r10 -40) = r1 7: (7b) *(u64 *)(r10 -48) = r1 8: (7b) *(u64 *)(r10 -56) = r1 9: (7b) *(u64 *)(r10 -64) = r1 10: (7b) *(u64 *)(r10 -72) = r1 11: (7b) *(u64 *)(r10 -80) = r1 12: (7b) *(u64 *)(r10 -88) = r1 13: (7b) *(u64 *)(r10 -96) = r1 14: (7b) *(u64 *)(r10 -104) = r1 15: (7b) *(u64 *)(r10 -112) = r1 16: (7b) *(u64 *)(r10 -120) = r1 17: (7b) *(u64 *)(r10 -128) = r1 18: (7b) *(u64 *)(r10 -136) = r1 19: (7b) *(u64 *)(r10 -144) = r1 20: (7b) *(u64 *)(r10 -152) = r1 21: (7b) *(u64 *)(r10 -160) = r1 22: (7b) *(u64 *)(r10 -168) = r1 23: (7b) *(u64 *)(r10 -176) = r1 24: (7b) *(u64 *)(r10 -184) = r1 25: (7b) *(u64 *)(r10 -192) = r1 26: (7b) *(u64 *)(r10 -200) = r1 27: (7b) *(u64 *)(r10 -208) = r1 28: (7b) *(u64 *)(r10 -216) = r1 29: (7b) *(u64 *)(r10 -224) = r1 30: (7b) *(u64 *)(r10 -232) = r1 31: (7b) *(u64 *)(r10 -240) = r1 32: (7b) *(u64 *)(r10 -248) = r1 33: (7b) *(u64 *)(r10 -256) = r1 34: (7b) *(u64 *)(r10 -264) = r1 35: (7b) *(u64 *)(r10 -272) = r1 36: (7b) *(u64 *)(r10 -280) = r1 37: (7b) *(u64 *)(r10 -288) = r1 38: (7b) *(u64 *)(r10 -296) = r1 39: (7b) *(u64 *)(r10 -304) = r1 40: (7b) *(u64 *)(r10 -312) = r1 41: (bf) r7 = r10 42: (07) r7 += -312 43: (bf) r1 = r7 44: (b7) r2 = 48 45: (bf) r3 = r6 46: (85) call bpf_probe_read#4 47: (79) r3 = *(u64 *)(r6 +24) 48: (bf) r1 = r10 49: (07) r1 += -256 50: (b7) r8 = 256 51: (b7) r2 = 256 52: (85) call bpf_probe_read_str#45 53: (bf) r1 = r0 54: (67) r1 <<= 32 55: (77) r1 >>= 32 56: (bf) r5 = r0 57: (07) r5 += 56 58: (2d) if r8 > r1 goto pc+1 R0=inv(id=0) R1=inv(id=0,umin_value=256,umax_value=4294967295,var_off=(0x0; 0xffffffff)) R5=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=fp-312,call_-1 R8=inv256 R10=fp0,call_-1 fp-264=0 59: (b7) r5 = 312 60: (63) *(u32 *)(r10 -264) = r0 61: (67) r5 <<= 32 62: (77) r5 >>= 32 63: (bf) r1 = r6 64: (18) r2 = 0xffff8b9120cc8500 66: (18) r3 = 0xffffffff 68: (bf) r4 = r7 69: (85) call bpf_perf_event_output#25 70: (b7) r0 = 0 71: (95) exit from 58 to 60: R0=inv(id=0) R1=inv(id=0,umax_value=255,var_off=(0x0; 0xff)) R5=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=fp-312,call_-1 R8=inv256 R10=fp0,call_-1 fp-264=0 60: (63) *(u32 *)(r10 -264) = r0 61: (67) r5 <<= 32 62: (77) r5 >>= 32 63: (bf) r1 = r6 64: (18) r2 = 0xffff8b9120cc8500 66: (18) r3 = 0xffffffff 68: (bf) r4 = r7 69: (85) call bpf_perf_event_output#25 R5 unbounded memory access, use 'var &= const' or 'if (var < const)' libbpf: -- END LOG -- libbpf: failed to load program 'syscalls:sys_enter_openat' libbpf: failed to load object 'tools/perf/examples/bpf/augmented_syscalls.c' bpf: load objects failed: err=-4007: (Kernel verifier blocks program loading) event syntax error: 'tools/perf/examples/bpf/augmented_syscalls.c' \___ Kernel verifier blocks program loading After: # perf trace -e tools/perf/examples/bpf/augmented_syscalls.c cat /etc/passwd > /dev/null 0.000 cat/29249 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC) 0.008 cat/29249 syscalls:sys_exit_openat:0x3 0.021 cat/29249 openat(dfd: CWD, filename: /lib64/libc.so.6, flags: CLOEXEC) 0.025 cat/29249 syscalls:sys_exit_openat:0x3 0.180 cat/29249 open(filename: /usr/lib/locale/locale-archive, flags: CLOEXEC) 0.185 cat/29249 syscalls:sys_exit_open:0x3 0.242 cat/29249 openat(dfd: CWD, filename: /etc/passwd) 0.245 cat/29249 syscalls:sys_exit_openat:0x3 # It also works with a more recent kernel: # uname -a Linux jouet 4.18.0-00014-g4e67b2a5df5d #6 SMP Thu Aug 30 17:34:17 -03 2018 x86_64 x86_64 x86_64 GNU/Linux # perf trace -e tools/perf/examples/bpf/augmented_syscalls.c cat /etc/passwd > /dev/null 0.000 cat/26451 openat(dfd: CWD, filename: /etc/ld.so.cache, flags: CLOEXEC) 0.020 cat/26451 syscalls:sys_exit_openat:0x3 0.039 cat/26451 openat(dfd: CWD, filename: /lib64/libc.so.6, flags: CLOEXEC) 0.044 cat/26451 syscalls:sys_exit_openat:0x3 0.231 cat/26451 open(filename: /usr/lib/locale/locale-archive, flags: CLOEXEC) 0.238 cat/26451 syscalls:sys_exit_open:0x3 0.278 cat/26451 openat(dfd: CWD, filename: /etc/passwd) 0.282 cat/26451 syscalls:sys_exit_openat:0x3 # Cc: Adrian Hunter Cc: Daniel Borkmann Cc: David Ahern Cc: Gianluca Borello Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Cc: Yonghong Song Link: https://lkml.kernel.org/n/tip-wkpsivs1a9afwldbul46btbv@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf augmented_syscalls: Check probe_read_str() return separately

2018-09-03T18:13:33Z

Using a value returned from probe_read_str() to tell how many bytes to copy using perf_event_output() has issues in some older kernels, like 4.17.17-100.fc27.x86_64, so separate the bounds checking done on how many bytes to copy to a separate variable, so that the next patch has only what is being done to make the test pass on older BPF validators. For reference, see the discussion in this thread: https://www.spinics.net/lists/netdev/msg480099.html Cc: Adrian Hunter Cc: Daniel Borkmann Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Cc: Yonghong Song Link: https://lkml.kernel.org/n/tip-jtsapwibyxrnv1xjfsgzp0fj@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf trace augmented_syscalls: Hook into syscalls:sys_exit_SYSCALL too

2018-08-30T18:52:21Z

Hook the pair enter/exit when using augmented_{filename,sockaddr,etc}_syscall(), this way we'll be able to see what entries are in the ELF sections generated from augmented_syscalls.c and filter them out from the main raw_syscalls:* tracepoints used by 'perf trace'. Cc: Adrian Hunter Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-cyav42qj5yylolw4attcw99z@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf trace augmented_syscalls: Rename augmented__syscall__enter to just _syscall

2018-08-30T18:52:21Z

As we'll also hook into the syscalls:sys_exit_SYSCALL for which there are enter hooks. This way we'll be able to iterate the ELF file for the eBPF program, find the syscalls that have hooks and filter them out from the general raw_syscalls:sys_{enter,exit} tracepoint for not-yet-augmented (the ones with pointer arguments not yet being attached to the usual syscalls tracepoint payload) and non augmentable syscalls (syscalls without pointer arguments). Cc: Adrian Hunter Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-cl1xyghwb1usp500354mv37h@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf augmented_syscalls: Update the header comments

2018-08-30T18:52:20Z

Reflecting the fact that it now augments more than syscalls:sys_enter_SYSCALL tracepoints that have filename strings as args. Also mention how the extra data is handled by the by now modified 'perf trace' beautifiers, that will use special "augmented" beautifiers when extra data is found after the expected syscall enter/exit tracepoints. Cc: Adrian Hunter Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-ybskanehmdilj5fs7080nz1g@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf trace augmented_syscalls: Augment sendto's 'addr' arg

2018-08-30T18:52:20Z

Its a 'struct sockaddr' pointer, augment it with the same beautifier as for 'connect' and 'bind', that all receive from userspace that pointer. Doing it in the other direction remains to be done, hooking at the syscalls:sys_exit_{accept4?,recvmsg} tracepoints somehow. Cc: Adrian Hunter Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-k2eu68lsphnm2fthc32gq76c@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf trace augmented_syscalls: Augment bind's 'myaddr' sockaddr arg

2018-08-30T18:52:20Z

One more, to reuse the augmented_sockaddr_syscall_enter() macro introduced from the augmentation of connect's sockaddr arg, also to get a subset of the struct arg augmentations done using the manual method, before switching to something automatic, using tracefs's format file or, even better, BTF containing the syscall args structs. # perf trace -e tools/perf/examples/bpf/augmented_syscalls.c 0.000 sshd/11479 bind(fd: 3, umyaddr: { .family: NETLINK }, addrlen: 12) 1.752 sshd/11479 bind(fd: 3, umyaddr: { .family: INET, port: 22, addr: 0.0.0.0 }, addrlen: 16) 1.924 sshd/11479 bind(fd: 4, umyaddr: { .family: INET6, port: 22, addr: :: }, addrlen: 28) ^C# Cc: Adrian Hunter Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-a2drqpahpmc7uwb3n3gj2plu@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

perf trace augmented_syscalls: Add augmented_sockaddr_syscall_enter()

2018-08-30T18:52:20Z

From the one for 'connect', so that we can use it with sendto and others that receive a 'struct sockaddr'. Cc: Adrian Hunter Cc: David Ahern Cc: Jiri Olsa Cc: Namhyung Kim Cc: Wang Nan Link: https://lkml.kernel.org/n/tip-8bdqv1q0ndcjl1nqns5r5je2@git.kernel.org Signed-off-by: Arnaldo Carvalho de Melo

user/sven/linux.git/tools/perf/examples, branch v4.20.5

perf augmented_syscalls: Start collecting pathnames in the BPF program

perf examples bpf: Start augmenting raw_syscalls:sys_{start,exit}

perf augmented_syscalls: Avoid optimization to pass older BPF validators

perf augmented_syscalls: Check probe_read_str() return separately

perf trace augmented_syscalls: Hook into syscalls:sys_exit_SYSCALL too

perf trace augmented_syscalls: Rename augmented_*_syscall__enter to just *_syscall

perf augmented_syscalls: Update the header comments

perf trace augmented_syscalls: Augment sendto's 'addr' arg

perf trace augmented_syscalls: Augment bind's 'myaddr' sockaddr arg

perf trace augmented_syscalls: Add augmented_sockaddr_syscall_enter()

perf trace augmented_syscalls: Rename augmented__syscall__enter to just _syscall