Master the Linux system logs with journalctl

journalctl is the command to read the system logs on any distribution that uses systemd. It replaces reading the files in /var/log/ by hand with a structured journal that you filter by service, by priority, by date and by boot. It is the main diagnosis tool of the Linux administrator.

What you will learn

• Read the system logs and the logs of a specific service
• Filter the logs by date, priority and boot
• Follow the logs in real time with -f
• Configure the retention and the maximum size of the journal
• Manage the permissions to read the logs
• Use the advanced output formats

In which context?

journalctl comes into play in every diagnosis situation, from a failed service to a disk filling up:

• A service refuses to start, the logs explain the error
• The system rebooted unexpectedly, read the previous boots
• An application produces intermittent errors, filter by priority
• The logs eat the disk space, check and configure the retention
• A security problem requires analysing the recent connections

LFCS / RHCSA

The EX200 objectives (RHEL 10) quote word for word "Locate and interpret system log files and journals" and "Preserve system journals". On the LFCS side, the journal serves the objective "Diagnose, identify, manage, and troubleshoot processes and services". You must be able to filter by service, by priority, by date and make the journal persistent.

The systemd journal

journald (systemd-journald) collects the logs from three sources, and knowing them tells you where a missing message could have been lost:

Source	Content
Kernel messages	Equivalent of dmesg
Syslog	Messages from services through the /dev/log socket
stdout/stderr	Output of the services managed by systemd

Swipe to see more

The journal is stored in binary form in /var/log/journal/ (persistent) or /run/log/journal/ (volatile, lost at reboot).

# Check the space used by the journal
journalctl --disk-usage

Archived and active journals take up 632.5M in the file system.

Journal persistence

Under the default setting Storage=auto, the journal becomes persistent as soon as /var/log/journal/ exists:

# Check that the directory exists
ls -ld /var/log/journal/

If the directory does not exist, the journal is volatile (lost at reboot). To enable persistence, you need four steps, and the last one is the one people forget:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald
sudo journalctl --flush

Without journalctl --flush, nothing moves

This is the missing step of almost every recipe found online. After the first three commands only, measured on AlmaLinux 10:

sudo find /var/log/journal -type f | wc -l

0

The directory is there, empty, and the service still announces Runtime Journal (/run/log/journal/...). It is journalctl --flush that asks journald to switch and copy the volatile journal to disk. Right after, the files appear and journalctl --header shows a File path under /var/log/journal/.

Another useful detail: systemd-tmpfiles --create --prefix /var/log/journal does not create the directory, contrary to what its name suggests. The rule that concerns it in /usr/lib/tmpfiles.d/journal-nocow.conf is an h, which sets a file attribute, not a d that would create the directory. The mkdir -p is therefore not optional.

The rule "the directory exists so the journal is persistent" only holds under Storage=auto. With Storage=volatile, the journal goes back to memory despite the presence of /var/log/journal/. The three values behave differently:

Storage=	Behaviour
auto (default)	Persistent if /var/log/journal/ exists, volatile otherwise
persistent	Persistent, the directory is created if needed
volatile	Always in memory, even if the directory exists

Swipe to see more

To stop depending on the existence of a directory, set Storage=persistent explicitly (see the Configuration section).

Read the logs

Without any filter, journalctl pours out the whole retained journal, from the oldest entry to the most recent, in a pager (less) that you quit with q. The options that follow reduce this flow to what interests you: a precise boot, a systemd unit, a time window. They add up in the same command, each extra option narrowing the result a little more.

Logs of the last boot

journald numbers the boots of the system: 0 is the current boot, -1 the previous one, and so on into the past. The -b option limits the output to a given boot, which isolates the messages of an incident without going through the whole history. --list-boots shows the retained boots with their identifier and time range; if this list holds only one entry under the header, the journal is not persistent.

# All the logs since the last boot
journalctl -b

# Logs of the previous boot
journalctl -b -1

# List the available boots
journalctl --list-boots

Excerpt measured on Ubuntu 24.04 (systemd 255), the two boots before the current one:

IDX BOOT ID                          FIRST ENTRY                  LAST ENTRY
 -2 6e42eed67c254fb1874f9cdb85918c14 Sat 2026-09-05 20:57:11 CEST Sun 2026-09-06 22:37:56 CEST
 -1 9254bc00425b4becac188df6401fd188 Sun 2026-09-06 22:38:28 CEST Mon 2026-09-14 08:53:02 CEST

Logs of a service

The -u option filters on a systemd unit, that is the name of the service as systemd knows it (nginx.service, which you can shorten to nginx). Repeating -u combines several units and shows their messages interleaved in chronological order, which helps to correlate a web server and its database during the same incident. If the output shows No entries, the unit name is almost always the cause.

# Logs of nginx
journalctl -u nginx

# Logs of nginx since the last boot
journalctl -u nginx -b

# Logs of several services
journalctl -u nginx -u postgresql

Follow the logs in real time

The -f option (follow) does not return: it shows new entries as the services write them, until Ctrl+C. It is the mode to start in a second terminal before restarting a service, to see the error at the precise moment it happens. The options combine, -fu nginx follows a single service live.

# Equivalent of tail -f
journalctl -f

# Follow a specific service
journalctl -fu nginx

Filter by date

--since and --until bound the search in time. They accept an absolute date in the YYYY-MM-DD HH:MM format as well as a relative expression in English (1 hour ago, 30 min ago, yesterday, today), interpreted against the current time of the machine. If you leave out --until, the upper bound is now. It is the fastest way to frame an incident whose approximate time you know.

# Since a date
journalctl --since "2026-06-28 08:00"

# Until a date
journalctl --until "2026-06-28 12:00"

# Relative
journalctl --since "1 hour ago"

# The last 30 minutes
journalctl --since "30 min ago"

# Yesterday
journalctl --since yesterday --until today

Filter by priority

The journal uses the syslog priority levels (0 = the most critical, 7 = the most verbose), and -p keeps a level and everything more severe than it:

Level	Name	Meaning
0	emerg	System unusable
1	alert	Immediate action required
2	crit	Critical error
3	err	Error
4	warning	Warning
5	notice	Normal but significant
6	info	Information
7	debug	Debug (very verbose)

Swipe to see more

# Errors and above (0 to 3)
journalctl -p err

# Errors since the last boot
journalctl -p err -b

# Warnings and above for a service
journalctl -u nginx -p warning

Priority in diagnosis

When a problem occurs, start with -p err -b to see only the errors of the current boot. If nothing shows up, widen to -p warning.

Search the logs

--grep applies a regular expression to the MESSAGE field of the entries, without piping into the grep command. The filtering happens while reading the binary journal, so faster, and the metadata of each entry stay usable. The search is case sensitive as soon as the pattern contains an uppercase letter, and insensitive otherwise; --case-sensitive=no forces insensitive matching in every case.

# Text search (built-in grep)
journalctl -b --grep "error|fail"

# Pattern with an uppercase letter, but matching forced case insensitive
journalctl -b --grep "Timeout" --case-sensitive=no

Kernel logs

The -k option (alias --dmesg) restricts the output to the messages emitted by the kernel: hardware detection at boot, device errors, out of memory, packets dropped by the firewall. Where dmesg numbers its lines in seconds since boot, journalctl stamps them with the real date and time, which lets you cross-check them with the application logs. Adding -b targets the current boot, essential after a hardware incident.

# Equivalent of dmesg with timestamps
journalctl -k

# Kernel messages of the current boot
journalctl -k -b

Output formats

journalctl offers several formats with the -o option, from the classic syslog line to JSON for a collection tool:

Format	Use
short	Default format (similar to syslog)
short-precise	With microseconds
verbose	Every structured field
json	JSON (for automated parsing)
json-pretty	Indented JSON (human reading)
cat	Only the message (no metadata)

Swipe to see more

# Verbose format (every field); the service is called crond on RHEL
journalctl -u cron -n 1 -o verbose

Excerpt measured on Ubuntu 24.04 (systemd 255): an entry carries many more fields, only the most useful ones are kept here, and the identifiers of the header are shortened.

Wed 2026-09-30 05:17:01.633361 CEST [s=891bfc9a…;i=344b68;b=0dbd62a1…;m=13…]
    PRIORITY=6
    _TRANSPORT=syslog
    SYSLOG_FACILITY=10
    SYSLOG_IDENTIFIER=CRON
    _COMM=cron
    _SYSTEMD_UNIT=cron.service
    MESSAGE=pam_unix(cron:session): session closed for user root
    _PID=2760268

The fields prefixed with an underscore (_PID, _COMM, _SYSTEMD_UNIT) are added by journald itself: the program that writes cannot forge them. The others, such as MESSAGE or SYSLOG_IDENTIFIER, come from the sender. The filters rely on these fields: journalctl _PID=2760268 or journalctl _COMM=cron work directly.

# JSON format for a centralisation tool
journalctl -u nginx -n 5 -o json

Log centralisation

The JSON format eases sending the logs to centralisation solutions such as Loki, Elasticsearch or Graylog. Use journalctl -o json combined with a collection agent.

journald configuration

The main file is not in the same place depending on the distribution. On Debian 13 and Ubuntu 24.04, /etc/systemd/journald.conf exists, with all its options commented out. On AlmaLinux 10, so RHEL 10, it does not exist: the shipped file is /usr/lib/systemd/journald.conf. In both cases, the safe method is the same, drop in a drop-in:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo nano /etc/systemd/journald.conf.d/99-retention.conf

Creating /etc/systemd/journald.conf masks the shipped file

systemd reads a single main file, the first one found, and /etc/ comes before /usr/lib/. Opening /etc/systemd/journald.conf in an editor on AlmaLinux 10 creates a blank file that fully masks the one of the distribution. Measured: the latter carries only one active line, Audit=, which leaves kernel auditing in its current state. Masked, it gives way to the compiled value, Audit=on according to the journald.conf(5) manual of systemd 257: journald turns on kernel auditing at start-up, without a single option having been written.

A drop-in under /etc/systemd/journald.conf.d/ does not carry this risk: it only overrides the options it declares, and the rest keeps coming from the file of the distribution. Debian and Ubuntu actually ship one, /usr/lib/systemd/journald.conf.d/syslog.conf. To see the effective configuration, all layers merged:

sudo systemd-analyze cat-config systemd/journald.conf

Essential options

The file follows the INI format: the [Journal] section first, then one option per line. A line commented out with # leaves the option at its default value compiled into systemd. The settings below cover the four decisions to make on a server: where to store the journal, what size to grant it, how long to keep it, and at what rate to accept the messages of a chatty service.

[Journal]
# Persistence of the journal (persistent, volatile, auto, none)
Storage=persistent

# Maximum size of the journal
SystemMaxUse=500M

# Maximum size per file
SystemMaxFileSize=50M

# Maximum retention time
MaxRetentionSec=3month

# Compression of the old files
Compress=yes

# Rate limiting, here the default values (messages per interval and per service)
RateLimitIntervalSec=30s
RateLimitBurst=10000

Option	Effect
Storage=persistent	The journal survives reboots
SystemMaxUse	Maximum total size of the journal
SystemMaxFileSize	Maximum size of each journal file
MaxRetentionSec	Maximum retention time
RateLimitBurst	Maximum number of messages per interval (per service)

Swipe to see more

After a change, restart and not reload:

sudo systemctl restart systemd-journald

systemd-journald cannot reload on the fly, and says so:

Failed to reload systemd-journald.service: Job type reload is not applicable
for unit systemd-journald.service.

A typo in the configuration only shows at service restart

Neither the editor nor systemd-analyze cat-config validates what you write. cat-config displays a misspelled key without blinking and exits with 0.

It is the restart that reveals the problem, and only in the journal:

sudo journalctl -u systemd-journald --since "1 min ago"

/etc/systemd/journald.conf.d/98-test.conf:2: Unknown key 'Compresss' in section
[Journal], ignoring.
/etc/systemd/journald.conf.d/98-test.conf:3: Failed to parse
Storage=persistant, ignoring: Invalid argument

Note the second message: persistant with an a instead of persistent is ignored, and the journal falls back to its default behaviour. Get into the habit of rereading the journal of the service right after each restart.

SystemKeepFree can cancel your SystemMaxUse

The classic trap of this section: SystemMaxUse sets a ceiling, but SystemKeepFree reserves an amount of free space on the file system, and the stricter of the two constraints wins.

Measured on a root offering 16,434 MB free, with SystemMaxUse=200M and SystemKeepFree=16500M:

System Journal (/var/log/journal/4a8981f6...) is 16M, max 16M, 0B free.

The effective ceiling drops to 16 MB, while the configuration asked for 200. Without the SystemKeepFree line, the same machine announces max 1.8G. If your journal stays stubbornly tiny, this is the first thing to check.

Rate limiting

If a service writes too many logs too fast, journald drops some. The Suppressed N messages message shows up in the logs. The default is 10,000 messages in 30 seconds per service, a ceiling journald multiplies according to the free disk space (journald.conf(5) manual, systemd 257). Raise RateLimitBurst if the behaviour is legitimate, or fix the chatty service.

Clean up the journal

The --vacuum-* options delete archived journal files, never the active file being written. The cleanup is therefore immediate but partial: if a single big active file takes the space, force a rotation first with journalctl --rotate. These commands need root rights and have no lasting effect, the journal will grow again as long as SystemMaxUse is not set in the configuration.

Reduce by size

--vacuum-size deletes the oldest archived files until the journal takes at most the requested size. The command lists each deleted file then announces the space freed, which leaves you a trace of what you have just permanently lost.

# Reduce the journal to 200 MB
sudo journalctl --vacuum-size=200M

Reduce by age

--vacuum-time reasons on the date of the entries and not on the volume: any archived file whose last entry is older than the given delay is deleted. The accepted units go from the second to the year (s, m, h, days, weeks, months, years). It is the variant to prefer when a retention policy imposes a keeping period rather than a size.

# Delete the logs older than 2 weeks
sudo journalctl --vacuum-time=2weeks

Reduce by number of files

--vacuum-files keeps only the N most recent journal files, without considering their size or their age. This criterion only makes sense if the files have a consistent size, so if SystemMaxFileSize is set. The three --vacuum-* options can appear in the same command, the strictest constraint wins.

# Keep only the last 5 files
sudo journalctl --vacuum-files=5

# Check the space after cleanup
journalctl --disk-usage

Permissions

The full journal is readable by root and by the systemd-journal group, owner of the files. On top of that come ACLs set by systemd-tmpfiles at boot, measured in /usr/lib/tmpfiles.d/systemd.conf: the adm group on Debian 13, Ubuntu 24.04 and AlmaLinux 10, plus the wheel group on AlmaLinux 10. For an account that must only read the logs, the dedicated group remains the right choice:

# Add a user to the group
sudo usermod -aG systemd-journal my-user

Unprivileged users only see the logs of their own services (user timers and services).

Journal and security

Two journald.conf settings concern using the journal in an audit context. The first makes any later change to the journal detectable, including by an attacker who obtained root rights. The second keeps feeding an existing syslog chain, for example to ship the messages to a remote collector the attacker does not control.

Check integrity

journald can seal the persistent files to make any tampering detectable. The Seal= option is already on by default (journald.conf(5) manual, systemd 257), but it does nothing without a sealing key: it is journalctl --setup-keys that creates it. The command produces a key that stays on the machine and a verification key to keep elsewhere, otherwise a root attacker could re-sign everything.

# Create the key pair (persistent journal required)
sudo journalctl --setup-keys

# Later, check the integrity with the key kept off the machine
sudo journalctl --verify --verify-key=<verification key>

Forward to syslog

To run journald alongside a traditional syslog server (rsyslog, syslog-ng), the ForwardToSyslog= option forwards every message to the syslog socket. It is off by default upstream (journald.conf(5) manual, systemd 257), but Debian 13 and Ubuntu 24.04 already turn it on through their syslog.conf drop-in. On AlmaLinux 10, which ships none, you set it yourself:

/etc/systemd/journald.conf.d/50-syslog.conf

[Journal]
ForwardToSyslog=yes

Troubleshooting

The symptoms below cover almost every problem met with the journal: nothing left after a reboot, service not found, dropped messages, full disk. Read the Probable cause column before applying the solution, several symptoms look alike without sharing the same origin. Two commands settle most cases: journalctl --list-boots to know whether the journal is persistent, and systemctl status systemd-journald to check that the daemon runs.

Symptom	Probable cause	Solution
Empty journal after reboot	Persistence not configured	Create /var/log/journal/, Storage=persistent, then journalctl --flush
/var/log/journal/ exists but the journal stays volatile	journalctl --flush forgotten, or Storage=volatile	Run --flush; check systemd-analyze cat-config systemd/journald.conf
Journal capped well below SystemMaxUse	SystemKeepFree reserves more free space than available	Read the is …, max …, … free line at service start
Setting without effect, no visible error	Misspelled key, silently ignored	journalctl -u systemd-journald right after the restart
No entries for a service	Wrong unit name	Check with systemctl list-units | grep name
Suppressed N messages	Rate limiting active	Raise RateLimitBurst or fix the service
Journal too big	No limit configured	Configure SystemMaxUse and MaxRetentionSec
User cannot see the logs	Not in the group	sudo usermod -aG systemd-journal user
Logs of previous boots missing	Volatile journal	Enable Storage=persistent

Swipe to see more

# Quick diagnosis
journalctl --disk-usage                    # Space used
journalctl --list-boots                     # Available boots
journalctl -p err -b                        # Errors of the current boot
systemctl status systemd-journald           # State of the daemon

Best practices

These reflexes avoid the two most frequent troubles with journald: a journal emptied at reboot just when you needed it, and a /var partition filled by logs without limit. They are set when the server is installed, not during the incident, because none of these settings rebuilds the entries already lost.

• Always enable persistence: without /var/log/journal/, the logs are lost at reboot
• Configure SystemMaxUse: keep the journal from filling the disk (200-500 MB is a good compromise)
• Configure MaxRetentionSec, 1 to 3 months depending on compliance needs
• Start the diagnosis with journalctl -p err -b: the errors of the current boot are the starting point
• Use -u service to target a service, do not go through the whole journal
• Schedule the cleanup, journalctl --vacuum-size or --vacuum-time regularly
• Centralise the logs in production, journald alone is not enough for an audit or a multi-server correlation

Practice

A journal that disappears at reboot can only be observed by really rebooting. This lab gives you a VM with a volatile journal: you enable Storage=persistent, create /var/log/journal/ and prove after a reboot that the previous boots stay readable with journalctl --list-boots.

Guided lab Practice

Learn how to do it

Make the systemd journal persistent across reboots

$ dsoxlab run l3-journald-persist

Environment: VM·Duration: ~20 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start l3-journald-persist checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/linux-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run l3-journald-persist
dsoxlab check l3-journald-persist

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key points

• journalctl reads the binary journal of systemd, it is the main diagnosis tool on Linux
• Filter by service (-u), by priority (-p), by date (--since) and by boot (-b) to target the useful information
• Persistence needs /var/log/journal/, Storage=persistent, and above all journalctl --flush that everybody forgets
• On RHEL and AlmaLinux, /etc/systemd/journald.conf does not exist: go through a drop-in under journald.conf.d/, otherwise you mask the shipped file (and its Audit= line)
• SystemKeepFree can cancel SystemMaxUse: the stricter constraint wins
• Priorities go from 0 (emerg) to 7 (debug), start with -p err when a problem occurs
• SystemMaxUse and MaxRetentionSec control the size and the retention of the journal
• journalctl --vacuum-size and --vacuum-time clean up by hand
• The JSON format (-o json) eases centralisation to Loki, Elasticsearch or Graylog

Knowledge check

Check that the essentials of this guide stuck. The questions cover only what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →