GitLab CI/CD pipeline reliability

Does your pipeline fail one time out of five for no apparent reason? Network timeout, unstable test, unavailable Docker registry, transient errors are the daily life of CI/CD pipelines. This guide gives you the mechanisms to absorb passing errors without hiding the real problems.

A reliable pipeline does not mean a pipeline that never fails. It is a pipeline that fails for the right reasons: a real bug, a real code problem, not a network glitch at 3 a.m.

What you will learn

Configure retries targeted by error type (retry:when)
Set timeouts adapted to each job
Detect and isolate flaky tests
Make a job idempotent (rerunnable without side effects)
Distinguish a transient error, a code error and an infrastructure error

Prerequisites

This lesson assumes you can read a failing pipeline: open a job's log, find the line where the script fails, and tell a code error from an infrastructure failure, without which no retry setting makes sense. It also assumes you know the execution rules (rules:), because the final example combines a workflow: with a manually started deployment, and that you have already written a multi-job .gitlab-ci.yml.

On the rights and tooling side, no paid edition is needed: the keywords presented here are written in the repository file, so you need to be able to edit it, which a Developer role allows. The two project settings mentioned (the project timeout and the automatic cancellation of redundant pipelines) live under Settings > CI/CD, which requires the Maintainer role on the project: with a Developer role you write the pipeline but you cannot open those settings. No local tool is required. Upstream lessons:

First pipeline: write a .gitlab-ci.yml with stages and jobs
Debug: reading GitLab CI/CD logs: read a job log and isolate the cause of a failure
Execution conditions (rules): decide when a job runs
Quality reports: the JUnit report that flaky-test detection relies on
CI/CD workflows: the workflow:rules block that opens the full pipeline on this page

In which context?

Your pipeline runs 50 times a day. Out of these 50 runs, 3 to 5 fail randomly:

An npm install that times out because the registry is slow
An integration test that fails because the PostgreSQL service was not ready
A docker pull that returns a 503 from Docker Hub
A unit test that passes locally but fails 1 time out of 10 in CI

Each transient failure triggers a manual investigation, a re-run, and frustration. This guide turns these failures into automatic recoveries.

Targeted retry

Rerunning a failed job is the most immediate reflex, and the most misused. A blind retry turns one clean failure into three slow failures, and sometimes hides a real defect behind a run that eventually passes. GitLab lets you restrict the rerun to certain categories of errors, which completely changes the cost/benefit ratio. The value of max: is capped at 2 retries, that is three executions in total.

Simple retry

The most basic mechanism: rerun a job on failure.

test:
  script: npm test
  retry: 2  # Up to 2 retries (3 executions max in total)

Problem: this retry catches all errors. A real bug in the tests will be rerun 2 times for nothing, wasting time.

Retry by error type (`retry:when`)

The best practice: target transient errors only.

build:docker:
  script:
    - docker build -t myapp .
    - docker push registry.example.com/myapp:$CI_COMMIT_SHA
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_pending_with_matching_runners
      - scheduler_failure

Available error types

These values correspond to the failure reason recorded by GitLab, visible on the job page. The row to remember is script_failure: it is the non-zero exit code of your own script, so in almost all cases a real defect that no rerun will fix.

Two values in this table are deprecated. The GitLab YAML reference declares stuck_or_timeout_failure and job_execution_timeout deprecated since GitLab 19.1. Each one was split into finer-grained reasons, listed in the table. For stuck_or_timeout_failure, the reference adds that since GitLab 19.0 this reason is kept for historical data only: new jobs carry one of the finer reasons, so configuring it alone no longer retries anything. You will come across these values in existing pipelines, so do not write them in your own.

`when` value	Meaning	Retry recommended?
`always`	Any error (default)	No, too broad
`unknown_failure`	Unclassified error	Yes
`script_failure`	The script returned a non-zero code	No, it is often a real bug
`api_failure`	GitLab API error	Yes
`stuck_pending_with_matching_runners`	Job pending for too long although matching runners exist	Yes
`stuck_pending_no_matching_runners`	Job pending because no runner matches its tags	No, configuration error
`no_updates_running` and `no_updates_canceling`	Job that no longer shows any sign of life, while running or while being canceled	Depends on the context
`stuck_or_timeout_failure`	Deprecated since GitLab 19.1: grouped the four previous reasons	No, use the values above
`runner_system_failure`	The runner crashed	Yes
`runner_unsupported`	Incompatible runner	No, not transient
`stale_schedule`	Stale schedule	No
`server_timeout_running` and `server_timeout_canceling`	The server cut the job for exceeding its time limit, while running or while being canceled	Depends on the context
`job_execution_timeout`	Deprecated since GitLab 19.1: grouped the two previous reasons	No, use the values above
`archived_failure`	Archived project	No
`unmet_prerequisites`	Unmet prerequisites	No
`scheduler_failure`	Scheduling error	Yes
`data_integrity_failure`	Data integrity	Yes

The stuck_pending_no_matching_runners value is deliberately absent from the examples on this page: a job that waits for lack of a matching runner reveals a configuration error (tags, protected branch) that a rerun will not fix. A targeted retry stays reserved for infrastructure failures.

Recommended patterns by job type

The number of retries is tuned according to the cost of one execution and the likelihood that the error is passing. A Docker build depends on remote resources and tolerates two retries; a deployment, for its part, must stay framed by a resource_group.

build:docker:
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_pending_with_matching_runners
      - unknown_failure

Docker errors are often linked to the network (pulling base images) or to the registry.

test:integration:
  services:
    - postgres:16-alpine@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685
  script:
    - npm run test:integration
  retry:
    max: 1
    when:
      - stuck_pending_with_matching_runners
      - runner_system_failure

A single retry, limited to infrastructure failures (lost runner, job left pending although runners exist). Real test failures (script_failure) are not retried, including a service that is not ready in time: that timing defect is fixed with an explicit wait (see the wait script in Services and advanced cache).

deploy:
  script: ./deploy.sh
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_pending_with_matching_runners
  resource_group: production

The resource_group guarantees that only one deployment runs at a time, even with retry.

Timeouts

Without an explicit limit, a job inherits the project timeout, set to 60 minutes by default. An npm install that waits indefinitely for an unreachable registry therefore occupies a runner for an hour before giving up. A tight timeout turns that wait into a fast, readable failure, and frees the execution queue for other pipelines.

Per-job timeout

Each job can have its own timeout:

lint:
  script: npm run lint
  timeout: 5 minutes  # Lint should not take more than 5 min

test:unit:
  script: npm test
  timeout: 15 minutes

test:e2e:
  script: npm run test:e2e
  timeout: 30 minutes  # End-to-end tests are longer

build:docker:
  script: docker build .
  timeout: 20 minutes

Global project timeout

Configurable in Settings > CI/CD > General pipelines > Timeout (default: 60 minutes).

The job timeout takes precedence over the project one: the GitLab YAML reference states that it can be longer than the project timeout, but not longer than the runner timeout. If the project is at 30 minutes, the job at 45 and the runner at 60, the job is cut at 45.

Choosing the right timeout

These values are starting points to compare with your own execution times, visible in the job history. Only one rule is universal: a timeout far wider than the real duration protects against nothing.

Job type	Recommended timeout	Reason
Lint	3-5 min	Very fast, a long timeout hides a problem
Unit tests	10-15 min	Depends on the number of tests
Integration tests	15-30 min	Services to start + tests
Docker build	10-20 min	Depends on the image and the cache
Deployment	10-15 min	If it takes longer, there is a problem
E2E tests	20-45 min	The longest, depending on the suite

Handling flaky tests

A flaky (unstable) test sometimes passes and sometimes fails, with no code change. Its real cost is not the time lost rerunning the pipeline: it is the loss of trust in the result. Past a certain instability threshold, the team reruns systematically without reading the error, and a real defect ends up going unnoticed.

Detecting flaky tests with GitLab

GitLab automatically detects flaky tests if you use JUnit reports:

test:
  script: npm test -- --reporters=jest-junit
  artifacts:
    reports:
      junit: junit.xml

GitLab flags as flaky a test that fails then passes (or the reverse) on the same MR without a code change. The results are visible in Analytics > CI/CD Analytics > Test cases.

Strategies for flaky tests

The three approaches below are not mutually exclusive: quarantine unblocks the pipeline immediately, test-level retry reduces noise, and only the fix solves the problem. Treat the first two as temporary measures with a deadline, otherwise the unstable test suite keeps growing.

Isolate flaky tests in a separate suite with allow_failure:

test:stable:
  script: npm test -- --testPathIgnorePatterns='flaky'
  # These must pass: a failure is a real bug

test:flaky:
  script: npm test -- --testPathPattern='flaky'
  allow_failure: true
  # To be watched: does not block the pipeline

Some test frameworks (Jest, pytest, RSpec) support native retry:

# Jest with jest-circus
test:
  script: npm test -- --bail --retries=2

# pytest with pytest-rerunfailures
test:
  script: pytest --reruns 2 --reruns-delay 1

Advantage: the retry applies to the individual test, not to the whole job. A flaky test that passes on the 2nd attempt does not affect the others.

The real solution: fix the test.

Frequent causes of flakiness:

Timing: sleep(1) in a test → use explicit waits
Order: tests that depend on the execution order → isolate the state
Shared resources: database not cleaned → transaction rollback
Date/time: a test that fails at midnight → mock the time

`allow_failure` for non-blocking jobs

A job with allow_failure: true runs and displays its result, but its failure does not stop the pipeline: the interface flags it with a warning rather than an error. The exit_codes: variant restricts this tolerance to specific exit codes, which lets you tell a warning from a blocking error within the same tool.

# The pipeline passes even if this job fails
security:scan:
  script: trivy image "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA"
  allow_failure: true  # Informational, does not block

# Variant: allow_failure only for specific exit codes
lint:strict:
  script: npm run lint:strict
  allow_failure:
    exit_codes:
      - 2  # Warnings = ok, Errors = blocking

Job idempotence

An idempotent job can be rerun without creating side effects. It is the basis of reliability: if the retry creates duplicates or inconsistent states, the remedy is worse than the disease.

Principles

These four rules are easy to check: rerun the same job twice on the same commit and compare the state of the target system. If it differs, the job is not idempotent and the retry becomes dangerous.

Principle	Example
No creation if it already exists	Check whether the Docker image exists before rebuilding
No global state modified	Use unique names based on `$CI_COMMIT_SHA`
Clean up before running	`rm -rf dist/ && npm run build`
Atomic operations	Deploy to a new directory then switch the symbolic link

Concrete examples

In the two pairs below, the fixed version applies the same idea: name the target after the commit, then switch in a single move. The deployment becomes replayable, and a failure along the way does not leave the application half-updated.

# ❌ Not idempotent: creates duplicates
deploy:
  script:
    - echo "$CI_COMMIT_SHA" >> /srv/deploy-history.log
    - cp -r dist/ /srv/app/

# ✅ Idempotent: the result is the same if rerun
deploy:
  script:
    - rm -rf /srv/app-$CI_COMMIT_SHA/
    - cp -r dist/ /srv/app-$CI_COMMIT_SHA/
    - ln -sfn /srv/app-$CI_COMMIT_SHA /srv/app

# ❌ Not idempotent: helm upgrade can fail if the release is in an intermediate state
deploy:k8s:
  script:
    - helm upgrade myapp ./chart --install

# ✅ Idempotent: forces the desired state
deploy:k8s:
  script:
    - helm upgrade myapp ./chart --install --atomic --timeout 5m
    # --atomic: automatic rollback on failure

`interruptible`: cancel obsolete jobs

When you push a new commit, the previous commit's jobs are often useless. interruptible lets GitLab cancel them automatically:

default:
  interruptible: true  # All jobs are cancelable by default

lint:
  script: npm run lint
  # interruptible: true (inherited from default)

test:
  script: npm test
  # interruptible: true (inherited from default)

deploy:production:
  script: ./deploy.sh
  interruptible: false  # NEVER interrupt a deployment in progress
  resource_group: production

To enable automatic cancellation: Settings > CI/CD > General pipelines > Auto-cancel redundant pipelines.

Complete pattern: a resilient pipeline

Here is a pipeline that combines all the techniques seen so far. The default: block carries the common settings, minimal retry and automatic cancellation, which each job can then override. Note the deliberate asymmetry between the tests, widely tolerant, and the production deployment: manual trigger, non-interruptible and serialized by resource_group.

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_COMMIT_TAG

default:
  interruptible: true
  retry:
    max: 1
    when:
      - runner_system_failure
      - stuck_pending_with_matching_runners

stages:
  - lint
  - test
  - build
  - deploy

lint:
  stage: lint
  timeout: 5 minutes
  script: npm run lint

test:unit:
  stage: test
  timeout: 15 minutes
  script: npm test -- --reporters=jest-junit --retries=1
  artifacts:
    reports:
      junit: junit.xml

test:integration:
  stage: test
  timeout: 20 minutes
  services:
    - postgres:16-alpine@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685
  script: npm run test:integration
  # retry inherited from default: a single retry, infrastructure failures

build:
  stage: build
  timeout: 15 minutes
  script:
    - rm -rf dist/
    - npm run build
  artifacts:
    paths: [dist/]

deploy:production:
  stage: deploy
  timeout: 10 minutes
  interruptible: false
  script: ./deploy.sh --atomic
  retry:
    max: 2
    when:
      - runner_system_failure
  resource_group: production
  rules:
    - if: $CI_COMMIT_TAG =~ /^v\d+/
      when: manual

Troubleshooting

The question to ask in front of an unstable pipeline is always the same: does the failure come from the code, from the infrastructure, or from the reliability configuration itself? The first row is a pure configuration error, which prevents any job from starting. The next two fall into the last case, where the mechanism meant to absorb errors becomes itself the cause of the problem.

Symptom	Likely cause	Solution
Pipeline failed as soon as it is created, no job displayed	A job is placed in a `stage:` that the `stages:` list does not declare: GitLab then rejects the whole configuration, with no message in the pipeline list	Validate the configuration with the project's CI linter (pipeline editor, validation tab), which names the missing stage; declare the stage in `stages:`
Job rerun but same error	`script_failure` included in retry (real bug)	Use a targeted `retry: when:`, exclude `script_failure`
Endless retry (3 attempts, same timeout)	Timeout too short for the job	Increase the timeout or optimize the job
Flaky test not detected by GitLab	No JUnit report	Add `artifacts: reports: junit:`
Pipeline not canceled after a new push	`interruptible` not configured	Add `default: interruptible: true`
Deployment interrupted halfway	`interruptible: true` on the deploy job	Explicitly set `interruptible: false`
Job that creates duplicates on retry	Non-idempotent job	Use unique names (`$CI_COMMIT_SHA`) and clean up beforehand

Hands-on lab

The lab "Reliable without hiding failures" puts this guide into practice on a GitLab CE forge of your own, set up on a virtual machine with its runner and its registry: no minute quota, no identity verification, and a project you can break as often as you need.

The pipeline is broken before it starts: a job sits in a stage that stages: does not declare, and no job runs at all. This symptom is not covered in the body of the lesson: the first row of the troubleshooting table above gives the approach to diagnose it. You repair it, then make it reliable: a retry limited to infrastructure failures, explicit timeouts, and a lock (resource_group) so two deployments never start together. The grade is based on the locks the forge really created, and it reads the default branch main: that is where you push.

The lab starts with dsoxlab run gl-18-fiabiliser-sans-masquer, after dsoxlab provision prepares the virtual machine, its runner and its registry. Installing DSOXLab and its prerequisites are described in the DSOXLab documentation.

Learn how to do it

Reliable without hiding failures

$ dsoxlab run gl-18-fiabiliser-sans-masquer

Environment: VM·Duration: ~40 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start gl-18-fiabiliser-sans-masquer checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/gitlab-ci-dsoxlab-training

Before you run the lab, check:

DSOXLab installed
catalogue available
compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run gl-18-fiabiliser-sans-masquer
dsoxlab check gl-18-fiabiliser-sans-masquer

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key takeaways

retry: when: targets transient errors, never retry script_failure except in specific cases
stuck_or_timeout_failure and job_execution_timeout are deprecated since GitLab 19.1: write the precise reasons (stuck_pending_with_matching_runners, server_timeout_running…), and never retry stuck_pending_no_matching_runners, which signals a configuration error
Timeouts must be adapted to each job type (lint 5 min, deploy 10 min)
Flaky tests are the first enemy of reliability, detect them with JUnit, fix them first
An idempotent job can be rerun without side effects, the key to reliable retry
interruptible: true cancels obsolete jobs, except deployments
Reliability does not come from a single mechanism but from the combination: targeted retry + adapted timeout + idempotence + interruptible
A reliable pipeline fails for the right reasons: a real bug, not a network glitch

Test your knowledge

These questions only cover the notions treated here: targeted retry, timeouts, unstable tests, idempotence and cancellation of obsolete jobs.

Knowledge check

Check what you have learned with this interactive quiz

10 questions

5 min

70% to pass

How it works

The timer starts when you click Start
Multiple choice, true/false and short answer questions
You can move between questions
Detailed results are shown at the end

GitLab CI/CD pipeline reliability

What you will learn

Prerequisites

In which context?

Targeted retry

Simple retry

Retry by error type (`retry:when`)

Available error types

Recommended patterns by job type

Timeouts

Per-job timeout

Global project timeout

Choosing the right timeout

Handling flaky tests

Detecting flaky tests with GitLab

Strategies for flaky tests

`allow_failure` for non-blocking jobs

Job idempotence

Principles

Concrete examples

`interruptible`: cancel obsolete jobs

Complete pattern: a resilient pipeline

Troubleshooting

Hands-on lab

Key takeaways

Test your knowledge

Knowledge check

How it works

Review

Score by domain

Summary

Strengths

Weak areas

Recommended next

GitLab CI/CD pipeline reliability

What you will learn

Prerequisites

In which context?

Targeted retry

Simple retry

Retry by error type (retry:when)

Available error types

Recommended patterns by job type

Timeouts

Per-job timeout

Global project timeout

Choosing the right timeout

Handling flaky tests

Detecting flaky tests with GitLab

Strategies for flaky tests

allow_failure for non-blocking jobs

Job idempotence

Principles

Concrete examples

interruptible: cancel obsolete jobs

Complete pattern: a resilient pipeline

Troubleshooting

Hands-on lab

Key takeaways

Test your knowledge

Knowledge check

How it works

Score by domain

Summary

Strengths

Weak areas

Recommended next

Retry by error type (`retry:when`)

`allow_failure` for non-blocking jobs

`interruptible`: cancel obsolete jobs