Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français

bpftrace: trace the Linux kernel in one line

By updated

Improve this page

bpftrace answers the question "what is my machine doing right now?" in one line. It is the most accessible eBPF tool: you describe what you want to observe in a short language, and it produces the eBPF program, has it verified, loads it and prints the result. This page covers the structure of the language, the probe types, data aggregation (counters, histograms), the ready-made BCC tools (diagnostic programs built on eBPF and shipped as a package) and the pitfalls, with real outputs captured on an Ubuntu 24.04 VM (kernel 6.8, bpftrace v0.20.2), completed by readings on AlmaLinux 10.2 for the BCC tools. For installation and the very first contact, see writing your first eBPF program.

What you will learn

  • Break down a bpftrace program: probe, filter, action
  • Choose the right probe type among the thousands available
  • Aggregate events into counters and histograms
  • Diagnose a machine with the ready-made BCC tools
  • Know when bpftrace is enough and when another tool is needed

How is a bpftrace program structured?

Every bpftrace program, from the one-liner to the script, follows the same three-part pattern:

probe /optional filter/ { action }
  • The probe is the trigger point: "when this event happens".
  • The filter (between /.../) is a condition: act only if it is true.
  • The action (between {...}) is what runs at each trigger.

A minimal example, already seen: print every program started on the machine. On Linux, starting a program goes through the execve system call.

Terminal window
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%-16s -> %s\n", comm, str(args->filename)); }'

Real output:

bash -> /usr/bin/uname
bash -> /usr/bin/id
bash -> /usr/bin/sleep

Here the probe is tracepoint:syscalls:sys_enter_execve, there is no filter, and the action prints two pieces of information. The variables comm (process name) and args->filename (argument of the call) are provided by bpftrace.

Which variables does bpftrace provide?

bpftrace exposes ready-to-use variables in every action; the table lists the most useful ones for diagnosis.

VariableContent
commName of the current process (16 characters)
pid / tidNumber of the process / of the thread
uidUser identifier
argsThe arguments of the probe (e.g. args->filename)
retvalThe return value (on exit probes)
nsecsTimestamp in nanoseconds

Which probe should you choose?

A probe designates where and when your action fires. bpftrace exposes a considerable number of them: on the test VM, sudo bpftrace -l counts more than 120,000 (124,308 at the last run). This total depends on the kernel and the loaded modules, so it varies from one machine to another. The main families:

  • tracepoint: a stable observation point, guaranteed by the kernel developers. Prefer it when it exists, because it does not change from one version to the next. Example: tracepoint:syscalls:sys_enter_openat.
  • kprobe / kretprobe: the entry / the exit of any kernel function. Very powerful, but tied to the internal names of the kernel, which can change. Example: kretprobe:vfs_read.
  • uprobe / uretprobe: the entry / the exit of a function of a user-space program, not only of the kernel.
  • interval / profile: a periodic trigger (every N seconds, or at a sampling interval), useful for summaries and profiling.

To discover the probes of a domain, filter the list:

Terminal window
sudo bpftrace -l 'tracepoint:syscalls:sys_enter_exec*'
tracepoint:syscalls:sys_enter_execve
tracepoint:syscalls:sys_enter_execveat

How do you aggregate: count and distribute?

Printing every event quickly becomes unreadable under load. The strength of bpftrace is to aggregate into maps, special variables prefixed with @. The kernel accumulates, and bpftrace prints only the summary at the end.

Counting events

How many system calls does each program make? We index a map by comm and increment a counter. The probe interval:s:5 { exit(); } stops the measurement after five seconds.

Terminal window
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); } interval:s:5 { exit(); }'

Real output (sorted by bpftrace, from least to most frequent):

@[systemd-journal]: 5
@[gmain]: 6
@[multipathd]: 31
@[bpftrace]: 115
@[bash]: 3424
@[sleep]: 3584
@[cat]: 3872
@[ls]: 4736
@[id]: 7040

In one command, you know which processes call the kernel the most. count() is an aggregation function; the kernel maintains a counter per key, without sending every event back to user space.

Distributing into a histogram

For a numeric quantity, a histogram is worth a thousand lines. What is the size of file reads? We hook the exit of vfs_read (the kernel function behind reads) and put retval (the number of bytes read) into a histogram with hist().

Terminal window
sudo bpftrace -e 'kretprobe:vfs_read /retval>0/ { @bytes = hist(retval); } interval:s:5 { exit(); }'

Real output:

@bytes:
[4, 8) 42 |@@@@ |
[8, 16) 1 | |
[16, 32) 21 |@@ |
[256, 512) 42 |@@@@ |
[512, 1K) 504 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1K, 2K) 21 |@@ |
[2K, 4K) 84 |@@@@@@@@ |

Each line is a size interval ([512, 1K) = from 512 to 1024 bytes), followed by the number of reads and a proportional bar. At a glance, the overwhelming majority of reads are between 512 and 1024 bytes. The filter /retval>0/ discards empty or failed reads.

How do you find out which process opens which files?

The two building blocks (probe + action) answer a common diagnostic question: which process touches which files? We trace file opening through openat.

Terminal window
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%-14s %s\n", comm, str(args->filename)); }'

Real output:

bash /dev/null
cat /etc/ld.so.cache
cat /lib/x86_64-linux-gnu/libc.so.6
cat /usr/share/locale/locale.alias
cat /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache

Without installing an agent or modifying cat, you see every library and every file it opens. This is the kind of observation that replaces hours of targeted strace, at the scale of the whole machine.

How do you trace a user-space function (uprobe)?

So far we have observed the kernel. A uprobe (user probe) does the same on a function of a user-space program: we hook a specific function of a binary, without having its source code or modifying it. Useful to understand the behavior of an application or a library.

Take a small test program, compiled to /tmp/prog, that calls a function compute() in a loop. On your machine, replace it with a binary and a function of your choice; sudo bpftrace -l 'uprobe:/path/to/binary:*' lists the functions that this binary exposes. Here we trace compute() and its first argument (arg0):

Terminal window
sudo bpftrace -e 'uprobe:/tmp/prog:compute { printf("compute() called with arg=%d\n", arg0); }'

Real output captured on the VM:

compute() called with arg=6
compute() called with arg=7
compute() called with arg=8
compute() called with arg=9

The probe uprobe:/tmp/prog:compute designates the binary then the function. You can also target a shared library (for example a function of libc) to observe every application that uses it. It is the user-space counterpart of the kernel kprobes.

Diagnosing with the ready-made BCC tools

For a first diagnosis, the administrator does not write a script: they run a BCC tool that is already written. The BCC package ships dozens of tools, maintained by the eBPF community, that answer the most common operations questions: who starts which process, who opens which file, what is the disk latency, how long a process waits before getting a CPU. This is the approach that Brendan Gregg, a systems performance specialist, describes in his book BPF Performance Tools: start with the existing tools, and write a one-liner only for the question they do not cover.

How do you install the BCC tools?

The package does not have the same name depending on the family, and above all the binaries have neither the same name nor the same location. On Debian and Ubuntu, each tool carries the -bpfcc suffix and is in the PATH. On RHEL and derivatives, the tools keep their short name but live under /usr/share/bcc/tools/, outside the PATH.

Terminal window
sudo apt install -y bpfcc-tools linux-headers-$(uname -r)
sudo execsnoop-bpfcc --help

The --help command must display the usage of the tool: it is the check that the package is installed and that the program loads. Verified in the lab on AlmaLinux 10.2 (bcc-tools 0.35.0): the tools do live under /usr/share/bcc/tools/, and their manual pages carry the bcc- prefix (man bcc-execsnoop). The examples in this section use the Debian/Ubuntu names; on RHEL, replace execsnoop-bpfcc with /usr/share/bcc/tools/execsnoop.

A trap awaits the installation on RHEL: dnf install kernel-devel pulls the most recent version of the package, not the one of the running kernel. If the machine has not rebooted since its last update, the two diverge, and the link /lib/modules/$(uname -r)/build points into the void. Compare uname -r and rpm -q kernel-devel: if they differ, install kernel-devel-$(uname -r) or reboot on the recent kernel. In the lab, BCC still worked thanks to the embedded headers of the kernel (kheaders), but do not count on this fallback.

Which tool for which question?

The table links the question you are asking to the tool that answers it. All of them load an eBPF program and therefore run with sudo. The CAP_BPF and CAP_PERFMON capabilities are not enough here: measured in the lab, bpftrace answers ERROR: bpftrace currently only supports running as the root user. to a non-root account holding these two capabilities, and opensnoop (BCC) fails on create_probe_event: open(/sys/kernel/tracing/kprobe_events): Permission denied (Ubuntu 24.04 and AlmaLinux 10.2). These capabilities remain the path of the agents designed to make do with them, not that of the tools on this page.

QuestionTool (Debian/Ubuntu)What it shows
Who starts which process?execsnoop-bpfccEvery new process, its parent and its arguments
Who opens which file, and fails?opensnoop-bpfccEvery open, the returned descriptor and the error
Is the disk slow?biolatency-bpfccThe distribution of disk I/O latencies (histogram)
Who connects where, for how long?tcplife-bpfccLifetime and volume of each TCP connection
Is the CPU saturated?runqlat-bpfccThe time processes wait before getting a CPU

Who starts which processes with execsnoop?

execsnoop-bpfcc prints every execve with the lineage of the process. It is the tool to run when a script "launches things" that you cannot explain, or when a cron job consumes CPU without you knowing why.

Terminal window
sudo execsnoop-bpfcc

Real output, captured on the Ubuntu 24.04 VM (BCC 0.29.1):

PCOMM PID PPID RET ARGS
uname 24635 24616 0 /usr/bin/uname -r
id 24636 24616 0 /usr/bin/id -u
sleep 24637 24616 0 /usr/bin/sleep 0.25

Each column speaks: PCOMM is the program name, PID its number, PPID that of its parent, RET the return code of the launch (0 = succeeded) and ARGS the full command. The PPID is the key to reading: three processes uname, id and sleep sharing the same PPID 24616 reveal a single script chaining them, not three users.

Which file fails to open, with opensnoop?

opensnoop-bpfcc prints every file open, with the result. Filter it on a process name with -n so as not to be drowned: useful when a service refuses to start without a clear message.

Terminal window
sudo opensnoop-bpfcc -n cat

Real output, captured on the AlmaLinux 10.2 VM (/usr/share/bcc/tools/opensnoop -n cat) while a second terminal ran cat /etc/hostname:

PID COMM FD ERR PATH
25205 cat 3 0 /etc/ld.so.cache
25205 cat 3 0 /lib64/libc.so.6
25205 cat 3 0 /etc/hostname

The FD column is the file descriptor obtained, and ERR the error code: 0 means success. The first two lines are the dynamic loader opening the library cache then libc, before cat finally opens the requested file. In diagnosis, the line you look for is the one where FD is -1 with a non-zero ERR: 2 (ENOENT) when the file does not exist, 13 (EACCES) when the process has no right to it. This is how a service that refuses to start without a clear message confesses the file it is missing.

Is the disk slow, with biolatency?

biolatency-bpfcc measures the time each disk I/O takes, from the moment the kernel issues it to the moment the device answers, and puts it in a histogram. The arguments 10 1 ask for a 10-second interval, once.

Terminal window
sudo biolatency-bpfcc 10 1

On AlmaLinux, the tool is outside the PATH: sudo /usr/share/bcc/tools/biolatency 10 1. Here is the real output on AlmaLinux 10.2 (kernel 6.12, BCC 0.35.0), while a dd if=/dev/zero of=/var/tmp/ddtest bs=1M count=300 oflag=direct wrote 300 MB to the VM's virtual disk:

Tracing block device I/O... Hit Ctrl-C to end.
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 0 | |
128 -> 255 : 0 | |
256 -> 511 : 1 |************* |
512 -> 1023 : 3 |****************************************|

On Ubuntu 24.04 (biolatency-bpfcc, BCC 0.29.1), the same measurement returned 2 I/Os in the 512 -> 1023 bucket. Each line is a latency bucket in microseconds (usecs) with the number of I/Os that fell into it, and the tool prints only up to the last non-empty bucket. Here, only 4 I/Os, all under a millisecond: this disk is virtual, so the absolute value cannot be compared to a real SSD. The gesture to remember is reading the shape: a single peak around a few hundred microseconds is typical of a healthy SSD. A second bump around several milliseconds signals I/Os that are queueing, therefore a saturated disk or a slow network volume, where iostat gives only an average that hides this tail.

When no tool covers the question, you move on to the bpftrace one-liners of the previous sections. And to write such a tool yourself, the optional lesson BCC and libbpf shows the developer's gesture.

When is bpftrace not the right tool?

bpftrace excels at one-off investigation: one question, one command, one answer, then you stop. It is not designed to run permanently as a monitoring service.

  • For continuous profiling (seeing at all times where the CPU is consumed, with a history), you need a dedicated collector such as Parca or Pyroscope, which belong to observability and not to this module.
  • To correlate events with Kubernetes (which pod, which container), a tool such as Inspektor Gadget is better suited (not yet translated).

Which version of bpftrace should you use?

This guide is based on the version provided by the distribution, that is v0.20.2 on Ubuntu 24.04 LTS. This is deliberate: a tool that loads code into the kernel is preferably taken from the distribution's repository, which tests it against the kernel it ships.

Upstream moves faster. At the time of the review, bpftrace publishes a v0.26.1 as a standalone binary in its GitHub releases, usable without compilation. Two surprises await whoever tries it, both verified on this Ubuntu 24.04 VM:

  • The binary refuses to start without privilege, with cannot write uid_map: Operation not permitted. It relies on unprivileged user namespaces, which Ubuntu restricts by default (kernel.apparmor_restrict_unprivileged_userns = 1). Under sudo, it works normally.
  • The binary of the v0.26.1 release announces itself as v0.26.0. The discrepancy comes from upstream, not from your installation. Do not rely on --version to know which release you downloaded.

The language and the probes of this page behave the same way on both versions. One cosmetic detail changes: 0.20 prints Attaching 1 probe... at startup, recent versions print Attached 1 probe. This is already the case of v0.24.2, shipped by AlmaLinux 10.2, on which five one-liners of this page (execve, bpftrace -l, count() per process, hist() on kretprobe:vfs_read, openat) were replayed: they run and produce outputs of the same format. The uprobe probe on a test binary was not replayed there.

Security and points of vigilance

  • Privileged operation. bpftrace loads eBPF programs: it requires root. Measured in the lab, bpftrace refuses to start under a non-root account, even one holding CAP_BPF and CAP_PERFMON. Reserve its use for administrators.
  • Observation, not modification. Tracing probes read the state of the system; they do not modify it. The risk is not breakage but information leakage: a probe can reveal file paths, command arguments, sensitive data. Treat its outputs as sensitive logs.
  • Cost under heavy load. A probe on a very frequent event (every system call) adds overhead. In production, filter early and prefer aggregation to line-by-line printing.
  • kprobe is not stable. A script based on kprobe:vfs_read can stop working if the kernel renames the function. Prefer a tracepoint when one exists.

Troubleshooting

This table covers the most frequent failures at first launch, of bpftrace as well as of the BCC tools.

SymptomLikely causeSolution
ERROR: Could not resolve symbolkprobe probe on a function absent from the kernelCheck the name with bpftrace -l, or use a tracepoint
No outputNo event, or filter too restrictiveTrigger some activity, loosen the filter
ERROR: bpftrace currently only supports running as the root user.bpftrace launched without sudoRelaunch with sudo
could not open bpf map: ..., error: Operation not permittedBCC tool launched without privilege (kernel.unprivileged_bpf_disabled is 2 by default)sudo (the CAP_BPF + CAP_PERFMON capabilities alone are not enough for opensnoop, measured in the lab)
Output redirected to a file incomplete or out of order (the PID COMM FD ERR PATH header appears after the first line)BCC buffers its output when it goes to a file; a total loss was seen in the lab, but the replay only showed the disorderPrefix with PYTHONUNBUFFERED=1 (header first, measured); the -d option sets a duration in seconds, it does not disable buffering
BCC fails to compile on RHELkernel-devel of a version other than uname -rInstall kernel-devel-$(uname -r), or reboot
execsnoop: command not found-bpfcc suffix forgotten (Debian/Ubuntu), or path outside PATH (RHEL)execsnoop-bpfcc, or /usr/share/bcc/tools/execsnoop
Truncated output / missing charactersString longer than the bufferIncrease with str(ptr, size)
Visible overhead on the machineProbe on an event that is too frequentFilter, aggregate, target a PID

Hands-on practice

On the lab VM, a hidden process writes to a hidden file every few seconds. You install the BCC tools (or bpftrace), you observe the kernel's openat and execve calls, then you name the program and the file. The validation compares your answer with the truth of the machine.

Key takeaways

  • A bpftrace program is a probe, an optional filter, an action.
  • Tracepoints are stable and preferred; kprobes are more powerful but tied to the internal names of the kernel.
  • Aggregation in @ maps (with count(), hist()) avoids drowning user space under events.
  • The BCC tools (execsnoop, opensnoop, biolatency, tcplife, runqlat) are the first diagnostic gesture; their names differ by family (-bpfcc or /usr/share/bcc/tools/).
  • bpftrace is made for one-off investigation, not for running as a permanent service.
  • It is a privileged operation whose outputs can be sensitive.

Knowledge check

Check that the essentials of this lesson are in place. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions
6 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer