Debug: reading GitLab CI/CD logs

Did your job fail with a cryptic message in the logs? This guide teaches you to read GitLab CI/CD logs efficiently and to quickly identify the cause of errors.

What you will learn

By the end of this guide, you will be able to:

• Find the cause of a failure: spot the ERROR line and the last command run in a log
• Tell the phases of a job apart: before_script, script, after_script, cache and artifacts
• Diagnose common errors: command or file not found, permission denied, failing tests, non-zero exit code
• Enable verbose mode with CI_DEBUG_TRACE, knowing the risk of leaking secrets
• Add your own traces and download or search the full logs

How are the logs structured?

A GitLab CI/CD job runs in several phases:

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Each section is collapsible in the interface. Click a section to see its content.

Variable section names

The exact section names can vary with the executor (Docker, Kubernetes, Shell), the runner version and certain features (distributed cache, etc.). The logical order stays similar.

How do you read the logs quickly?

1. Open the failed job

   Build > Jobs → click the red job

2. Go to the end of the logs

   Scroll down or click the "Scroll to bottom" button

3. Find the last command that ran

   Just before Job failed is the command that failed.

4. Read the error message

   Go up a few lines to see the complete error message.

What are the common errors and their solutions?

1. Command not found

$ npm run build
bash: npm: command not found

Cause: The tool is not installed in the Docker image.

Solutions:

• Use an image that contains the tool: image: node:24
• Install it in before_script: apt-get install -y nodejs npm

2. File not found

$ cat config.json
cat: config.json: No such file or directory

Cause: The file does not exist or the path is wrong.

Solutions:

• Check the path (absolute vs relative)
• Add ls -la beforehand to list the files
• Check that the artifacts are indeed downloaded

3. Permission denied

$ ./deploy.sh
bash: ./deploy.sh: Permission denied

Cause: The script is not executable.

Solution:

before_script:
  - chmod +x deploy.sh

4. Failed tests

FAIL src/utils.test.js
✕ should return correct value (5 ms)

Test Suites: 1 failed

Cause: The code does not pass the tests.

Solution: Fix the code or the tests!

5. Non-zero exit code

$ some_command
ERROR: Something went wrong
Job failed: exit code 1

Cause: The command returned an error code.

Diagnosis: Read the ERROR message above.

Debug mode: CI_DEBUG_TRACE

To see all the commands executed along with their values:

variables:
  CI_DEBUG_TRACE: "true"

job:
  script:
    - echo $MY_SECRET    # Will print the value!

Risk of leaking secrets

CI_DEBUG_TRACE exposes ALL the variables available to the job, including secrets and tokens. The logs become visible to everyone who has access to the job logs.

• Enable it only through "Run pipeline" + a temporary variable
• Check that access to the logs is restricted
• Delete the logs after analysis if the project is public

Enable it temporarily (without modifying the file):

1. Build > Pipelines > Run pipeline
2. Add the variable CI_DEBUG_TRACE = true
3. Run the pipeline

How do you add your own debug logs?

build:
  script:
    # Print environment variables
    - echo "Branch: $CI_COMMIT_BRANCH"
    - echo "Source: $CI_PIPELINE_SOURCE"

    # List files
    - ls -la
    - pwd

    # Print the content of a file
    - cat package.json

    # Your command
    - npm run build

How do you continue despite an error?

By default, the runner stops the job at the first command that fails (exit code ≠ 0). To continue despite an error:

job:
  script:
    - command_that_may_fail || true    # Continues even on error
    - another_command

Pitfalls with | and &&

Pipes: with set -o pipefail (enabled by default on some runners), cmd1 | cmd2 fails if any command in the pipe fails, even if the last one succeeds. A classic case is grep, which returns 1 when there is no match.

&& chains: cmd1 && cmd2 can hide errors if misused. Prefer commands on separate lines for predictable behavior.

To mark a job as "non-blocking" for the pipeline:

job:
  allow_failure: true
  script:
    - risky_command

allow_failure ≠ || true

allow_failure: true does not make the job "continue": the job still fails, but the pipeline continues (the job shows up in orange/warning instead of red). Use || true to ignore an error inside the job.

How do you download the complete logs?

To analyze offline or to share:

1. On the job page, use the Download button (to the right of the logs)
2. Or through the API:

# Replace with the URL of your GitLab instance
# The token must have the read_api scope on the project
curl --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/jobs/$JOB_ID/trace" \
  > job.log

What runs after a failure?

Even if the job fails or is canceled, after_script runs. Use it to collect information:

job:
  script:
    - npm test
  after_script:
    # Runs even if npm test fails or if the job is canceled
    - echo "Job status: $CI_JOB_STATUS"    # success, failed, or canceled
    - echo "Collecting debug info..."
    - cat npm-debug.log || true
    - ls -la node_modules/ || true

$CI_JOB_STATUS in after_script

During after_script, the $CI_JOB_STATUS variable contains the job status (success, failed, canceled). Useful to adapt the cleanup to the result.

How do you search the logs?

In the GitLab interface:

1. Ctrl+F (or Cmd+F) to search the page
2. Search for: error, failed, not found, denied

On the command line after downloading:

# Find the errors
grep -i "error\|failed\|denied" job.log

# With context (5 lines before/after)
grep -i -B5 -A5 "error" job.log

Runner logs vs job logs

This page covers the job logs (visible in the GitLab interface). If the job does not start or if you suspect an executor/cache problem, look at the runner logs:

# Logs of the gitlab-runner service
journalctl -u gitlab-runner -n 200

# If Docker executor
docker logs gitlab-runner

# If Kubernetes executor
kubectl logs -n gitlab-runner <pod-name>

Hands-on lab

The lab "Read a failure and fix it" replays this guide on a GitLab CE forge of your own, set up on a virtual machine with its runner and its registry, rather than on gitlab.com: no minute quota, no identity verification, and a project you can break as often as you need.

You diagnose a red pipeline from GitLab's own messages: first a configuration error that stops any job from running, then two code errors only a job log reveals. Grading is taken on the three fixes, read from the API. The two families of defect are not read in the same place, and that is the whole exercise: this page teaches you to read the log of a job, but a pipeline refused because of its configuration has no job, hence no log to read, and the pipeline page does not always give the cause. It is the CI Lint that names it, presented in Validate a .gitlab-ci.yml file.

The lab starts with dsoxlab run gl-02-lire-un-echec, after dsoxlab provision prepares the virtual machine, its runner and its registry. Installing DSOXLab and its prerequisites are described in the DSOXLab documentation.

Guided lab Practice

Learn how to do it

Read a failure and fix it

$ dsoxlab run gl-02-lire-un-echec

Environment: VM·Duration: ~30 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start gl-02-lire-un-echec checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/gitlab-ci-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run gl-02-lire-un-echec
dsoxlab check gl-02-lire-un-echec

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key takeaways

1. Scroll to the bottom of the logs to see the final error
2. exit code 1 = the command failed, read above it
3. Collapsible sections: click to open/close
4. CI_DEBUG_TRACE: true for verbose mode (warning: it exposes secrets!)
5. after_script runs even after a failure or a cancellation
6. allow_failure: true = the pipeline continues, but the job still fails
7. Pipes with |: beware of pipefail, which can make the line fail

Test your knowledge

Knowledge check

Check what you have learned with this interactive quiz

10 questions

5 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →