DAG and parallelism in GitLab CI/CD (needs)

Do your pipelines wait for a whole stage to finish when only one job is needed? With needs: (DAG), a job starts as soon as its real dependencies are done, without waiting for the whole stage. The result: pipelines that are often noticeably faster (and above all more parallel).

What is a DAG?

DAG = Directed Acyclic Graph. Behind this technical term lies a simple idea:

• Directed: dependencies have a direction (A → B means "B waits for A")
• Acyclic: no loops (A cannot wait for B if B already waits for A)

It is like a family tree: your parents gave birth to you, not the other way around. Dependencies go in one direction only.

Who is this guide for?

This guide is read by the section that matches your need. If your pipeline is too long because jobs wait needlessly, start with the problem without a DAG. If your dependencies are complex (a job depends on a single other job, not on a whole stage), see the configuration with needs. To visualize the graph of real dependencies in GitLab, read visualize the DAG. To avoid circular dependencies and other pitfalls, go to the common errors.

What you will learn

By the end of this module, you will be able to:

• Understand the problem: why stages slow down your pipelines
• Use needs: define the real dependencies between jobs
• Manage artifacts: artifacts: true/false with needs
• Visualize the DAG: read the dependency graph in GitLab
• Use needs: []: start a job immediately
• Avoid pitfalls: circular dependencies, missing jobs
• Recognize a stable cache key: one that survives from one commit to the next

Prerequisites

Before continuing, make sure you are comfortable with:

• GitLab CI/CD core concepts
• Artifacts and cache

The problem without a DAG

Default behavior: stage by stage

By default, GitLab runs pipelines stage by stage:

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: npm run build

lint:
  stage: test
  script: npm run lint

unit_tests:
  stage: test
  script: npm test

integration_tests:
  stage: test
  script: npm run test:integration

deploy:
  stage: deploy
  script: ./deploy.sh

Let us break down what happens:

1. Build stage: build runs alone (2 minutes)
2. Test stage: lint (30s), unit_tests (3 min), integration_tests (5 min) start in parallel
3. Forced wait: even if lint finishes in 30 seconds, the "test" stage lasts 5 minutes (the longest job)
4. Deploy stage: deploy finally starts

Problem: deploy waits for lint, unit_tests AND integration_tests to be all finished, even if deploy only needs unit_tests. If lint fails and you have to fix it, you waited 5 minutes for nothing.

In the diagram, the dashed arrows show what deploy waits for: the three jobs of the test stage, including integration_tests, which it does not actually need. This needless wait is what needs: removes.

Without a DAG: stages run one after the other. Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Configuration with needs

The needs: keyword changes the game: instead of saying "wait for the whole previous stage", you say "wait for only these specific jobs".

Basic syntax

needs: takes a list of job names. The current job starts as soon as all the listed jobs are done, whatever their stage.

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script: npm run build
  artifacts:
    paths:
      - dist/

lint:
  stage: test
  script: npm run lint
  needs: []                    # No dependency, starts immediately

unit_tests:
  stage: test
  script: npm test
  needs: ["build"]             # Waits only for build

integration_tests:
  stage: test
  script: npm run test:integration
  needs: ["build"]

deploy:
  stage: deploy
  script: ./deploy.sh
  needs: ["unit_tests"]        # Does not wait for lint or integration_tests

Let us analyze the changes:

Job	needs	What changes
lint	[] (empty)	Starts immediately, in parallel with build
unit_tests	["build"]	Waits only for build, not the whole stage
integration_tests	["build"]	Same, waits just for build
deploy	["unit_tests"]	Starts as soon as unit_tests finishes, without waiting for lint or integration_tests

Swipe to see more

Result: the diagram keeps only the real dependencies. The main path goes from build to deploy through unit_tests; lint and integration_tests, greyed out, run in parallel without delaying deploy.

With a DAG: each job waits only for what its needs lists. Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

The pipeline is faster because:

• lint starts immediately (no need to wait for build)
• deploy does not wait for lint or integration_tests
• If integration_tests takes 10 minutes but unit_tests only 2, deploy starts after 2 minutes

needs options

needs offers several syntaxes depending on your needs. Let us start with the simplest.

Simple list

The most common syntax: a list of job names between brackets.

deploy:
  needs: ["build", "test"]  # Waits for both build AND test to finish

The deploy job starts when both jobs are green. If one fails, deploy does not start (default behavior).

With artifact control

When you use needs, GitLab automatically downloads the artifacts of the referenced jobs (the files those jobs produced). But sometimes you want to wait for a job without fetching its files.

Concrete example: deploy waits for lint (to be sure the code is clean), but does not need the files produced by lint.

deploy:
  needs:
    - job: build
      artifacts: true      # Downloads the dist/ folder produced by build
    - job: lint
      artifacts: false     # Waits for lint, but downloads nothing

Why is it useful? Downloading artifacts takes time and bandwidth. If you do not need them, artifacts: false speeds up the start of the job.

Common pitfall: 'vanished' artifacts

As soon as a job uses needs, GitLab changes its behavior: it no longer automatically downloads the artifacts of the previous stages.

Without needs: the job receives the artifacts of all the jobs of the previous stages ("magic" behavior).

With needs: the job receives only the artifacts of the jobs listed in needs.

If your files "disappear" after you add needs, check that you listed the job that produces them.

needs: [], no dependency

This is the most powerful syntax for speeding up your pipelines. A job with needs: [] (empty list) starts immediately, without waiting for any other job.

lint:
  stage: test          # Even though it is in the "test" stage...
  needs: []            # ...it starts immediately, in parallel with "build"!
  script: npm run lint

Why does it work? Linting analyzes the source code, not the compiled code. It does not need to wait for build to produce the dist/ folder. So why make it wait?

Perfect jobs for needs: []

These jobs generally do not need the artifacts of the previous stages:

• Linting: ESLint, Pylint, golangci-lint
• Formatting: Prettier, Black, gofmt
• Static analysis: SonarQube (on the source code)
• Security: secret detection, SAST on the source code
• Documentation: link checking, docs generation

All these jobs can start at the very first moment of the pipeline.

needs with optional

Sometimes, a job in your needs may not exist in some pipelines. For example, security_scan only runs on the main branch (via rules). On a feature branch, this job does not exist.

Without optional: GitLab refuses to create the pipeline ("job not found").

With optional: true: GitLab silently ignores the missing job.

security_scan:
  stage: test
  rules:
    - if: $CI_COMMIT_BRANCH == "main"  # Only exists on main
  script: ./scan.sh

deploy:
  needs:
    - job: build                        # Required, must exist
    - job: security_scan
      optional: true                     # Optional: fine if the job does not exist

Behavior:

• On main: deploy waits for build AND security_scan
• On feature-x: deploy waits only for build (because security_scan does not exist)

Limits and special cases

The 50-job limit

By default, needs accepts a maximum of 50 jobs. Beyond that, GitLab refuses to create the pipeline. This limit exists to avoid overly complex pipelines that become hard to debug.

If you reach this limit, it is often a sign that your pipeline should be split into parent-child pipelines.

Interaction with parallel:

If you reference a job that uses parallel: (or parallel:matrix), your job waits for all the generated instances, not just one.

test:
  parallel: 3              # Generates test 1/3, test 2/3, test 3/3
  script: npm test

deploy:
  needs: ["test"]          # Waits for the 3 test jobs

To target a specific instance, see the guide on matrices.

Dependencies within the same stage

needs can reference jobs from the same stage. This is useful when two jobs are in the same stage but one depends on the other.

build_lib:
  stage: build
  script: npm run build:lib

build_app:
  stage: build
  needs: ["build_lib"]     # Same stage, but waits for build_lib
  script: npm run build:app

Do not mix needs and dependencies

dependencies is the old keyword for controlling artifacts. Do not use it in a job that already has needs: the behaviors can conflict.

needs:project (advanced)

needs:project lets you download artifacts from other pipelines or projects. It is useful for shared libraries, but with constraints: same GitLab instance, adequate permissions, and the branch/ref must be specified.

Visualize the DAG

GitLab displays the dependency graph in the interface:

1. Go to Build > Pipelines

2. Click on a pipeline

3. Select the Needs tab (or look at the graph that shows the arrows)

The graph shows the real dependencies between jobs, not the organization by stages.

Practical examples

Optimized Node.js pipeline

stages:
  - prepare
  - quality
  - build
  - test
  - deploy

install:
  stage: prepare
  script: npm ci
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  artifacts:
    paths:
      - node_modules/
    expire_in: 1 hour

# Quality jobs: no dependency, start immediately
lint:
  stage: quality
  needs: []
  script: npm run lint

format_check:
  stage: quality
  needs: []
  script: npm run format:check

# Build: waits only for install
build:
  stage: build
  needs: ["install"]
  script: npm run build
  artifacts:
    paths:
      - dist/

# Tests: in parallel, each waits only for what it needs
unit_tests:
  stage: test
  needs: ["install"]
  script: npm test

e2e_tests:
  stage: test
  needs: ["build"]           # Needs the build
  script: npm run test:e2e

# Deploy: waits for the critical tests, not the lint
deploy_staging:
  stage: deploy
  needs: ["unit_tests", "build"]
  script: ./deploy.sh staging

This pipeline saves time in two ways. The needs make lint and format_check start immediately, and deploy_staging waits only for what it depends on. The cache: block of install, for its part, relies on a key derived from package-lock.json: it changes only when the dependencies change, so the cache is found again from one commit to the next. A key that changes on every commit, like $CI_COMMIT_SHA, would do the opposite: the jobs of a single pipeline would share the cache, but the next pipeline would never find it. To check it, compare the key announced by the Checking cache for ... line of the log in two successive pipelines; the lesson Artifacts and cache details cache keys.

Monorepo with independent services

stages:
  - build
  - test
  - deploy

# Frontend and backend build in parallel
build_frontend:
  stage: build
  script: npm run build:frontend
  artifacts:
    paths:
      - frontend/dist/

build_backend:
  stage: build
  script: npm run build:backend
  artifacts:
    paths:
      - backend/target/

# Independent tests
test_frontend:
  stage: test
  needs: ["build_frontend"]
  script: npm run test:frontend

test_backend:
  stage: test
  needs: ["build_backend"]
  script: npm run test:backend

# Independent deployments
deploy_frontend:
  stage: deploy
  needs: ["test_frontend"]
  script: ./deploy-frontend.sh

deploy_backend:
  stage: deploy
  needs: ["test_backend"]
  script: ./deploy-backend.sh

The frontend and the backend are completely independent. If the backend is slow, the frontend still deploys.

Best practices

1. Ask yourself: "What does this job REALLY need?"

Before adding a job to needs, ask yourself: "If this job did not exist, could mine still run?" If so, do not add it.

Example: does deploy need to wait for lint? Probably not: lint checks code quality, but produces nothing that deploy needs.

2. Use needs: [] aggressively

Every job that can start immediately should have needs: []. It is the most powerful lever for speeding up your pipelines.

Simple rule: if the job needs no file produced by another job, set needs: [].

3. Disable useless artifacts

When you wait for a job for its status (not its files), use artifacts: false:

deploy:
  needs:
    - job: lint
      artifacts: false   # We just want lint to have passed, not its files
    - job: build
      artifacts: true    # We need the dist/ folder

4. Visualize the graph regularly

After changing your needs, look at the graph in GitLab (the "Needs" tab of the pipeline). It shows the real dependencies and helps you spot mistakes.

5. Keep stages for organization

Even with a DAG, stages remain useful for visual organization. Group jobs by "category" (build, test, deploy) even if their execution no longer strictly follows that order.

Common errors

Here are the most common problems and how to solve them.

1. Circular dependency

Symptom: the pipeline refuses to be created with "circular dependency detected".

Cause: two jobs reference each other, creating an impossible loop.

# ❌ FORBIDDEN: A waits for B, but B waits for A
job_a:
  needs: ["job_b"]

job_b:
  needs: ["job_a"]

Solution: rethink the architecture. Ask yourself: "Who really needs whom?" Often, one of the two needs is useless.

2. Too many dependencies

Symptom: "needs cannot have more than 50 entries".

Cause: you listed more than 50 jobs in a single needs.

Solution:

• Simplify: does this job really need 50 dependencies?
• Group: create an intermediate job that waits for several jobs, then depend on it
• Self-managed: the limit is configurable in the administrator settings

3. "Vanished" artifacts

Symptom: "file not found" even though the previous job did create them.

Cause: you added needs without including the job that produces the artifacts.

Reminder: with needs, you receive only the artifacts of the listed jobs. The "magic" behavior of the previous stages is disabled.

# ❌ Problem: deploy waits for lint, but does not have the artifacts of build
deploy:
  needs: ["lint"]
  script: ls dist/           # ERROR: dist/ does not exist!

# Solution: add build to needs
deploy:
  needs: ["lint", "build"]
  script: ls dist/           # OK: dist/ is downloaded from build

4. needs: [] but the job fails

Symptom: you set needs: [] to speed things up, but the job fails because files are missing.

Cause: the job actually needs artifacts from another job.

Solution: identify the missing files and add the job that produces them to needs.

# ❌ The job needs node_modules/, but waits for nobody
test:
  needs: []
  script: npm test           # ERROR: node_modules/ does not exist!

# Wait for the job that installs the dependencies
test:
  needs: ["install"]
  script: npm test           # OK: node_modules/ is available

Hands-on lab

The lab "A cache that is never found, and jobs that wait for nothing" puts this guide into practice on a GitLab CE forge of your own, set up on a virtual machine with its runner and its registry: no minute quota, no identity verification, and a project you can break as often as you need.

The pipeline works and wastes time in two ways. The first belongs to this lesson: its jobs queue behind stages that do not concern them, which needs: fixes. The second is its cache key, which changes on every commit, so the cache only serves inside the pipeline that created it and never from one commit to the next: the Node.js example above gives the principle. The grade is based on an identical cache key from one pipeline to the next, read in the logs after two runs, and on two jobs from different stages overlapping in time, read in their timestamps. It does not measure the duration itself, which varies with the load of the machine.

The lab starts with dsoxlab run gl-12-accelerer-le-pipeline, after dsoxlab provision prepares the virtual machine, its runner and its registry. Installing DSOXLab and its prerequisites are described in the DSOXLab documentation.

Guided lab Practice

Learn how to do it

A cache that is never found, and jobs that wait for nothing

$ dsoxlab run gl-12-accelerer-le-pipeline

Environment: VM·Duration: ~35 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start gl-12-accelerer-le-pipeline checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/gitlab-ci-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run gl-12-accelerer-le-pipeline
dsoxlab check gl-12-accelerer-le-pipeline

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key takeaways

1. needs defines the real dependencies, not the stages
2. needs: [] = no dependency, starts immediately
3. Artifacts: with needs, only those of the listed jobs are downloaded
4. Limit: max 50 jobs in needs by default
5. No loops, dependencies must be acyclic
6. Do not mix needs and dependencies in the same job
7. Visualize the graph in GitLab to validate

Test your knowledge

Knowledge check

Check what you have learned with this interactive quiz

10 questions

5 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →