GitLab CI/CD quality reports

Your tests pass but all you see is "Job succeeded"? GitLab can display detailed results directly in merge requests: failed tests, code coverage, quality issues. This guide shows you how to configure quality reports.

What you will learn

By the end of this module, you will know how to:

• Configure JUnit reports: display failed tests in the MR
• Extract code coverage: display the global % and the line-by-line diff
• Integrate quality reports: ESLint, SonarQube in Code Climate format
• Enable security reports: SAST, secrets, dependencies
• Use dotenv: pass variables between jobs
• Avoid common mistakes: when: always, report formats

Prerequisites

Before continuing, make sure you are comfortable with:

• Artifacts and cache
• Rules and conditions

Report types (artifacts:reports)

GitLab supports several types of reports, each with its own dedicated display: a report is not an ordinary artifact. A file declared under artifacts:paths is only stored, to be downloaded from the job; the same file declared under artifacts:reports is interpreted (parsed) by GitLab. That is what lets it display a result in the merge request interface instead of a file to download. In return, the expected format is not negotiable: an XML that is valid but follows another schema is ingested without error and displays nothing.

Type	Display	Format
junit	Failed tests in MR	JUnit XML
coverage_report	Line-by-line coverage diff	Cobertura XML
codequality	Quality issues in MR	Code Climate JSON
sast	Code vulnerabilities	GitLab JSON (SARIF has its own keyword, sarif)
secret_detection	Detected secrets	GitLab JSON
dependency_scanning	Dependency vulnerabilities	GitLab JSON
dotenv	Variables for following jobs	KEY=value

Swipe to see more

JUnit reports

The JUnit XML format is the common denominator of test frameworks: Jest, pytest, Go and Maven can all produce it, sometimes through a plugin. GitLab uses it to list failing tests with their error message, and to flag those that regress compared with the target branch. Without this report, a red pipeline forces you to open the job logs to find out what broke.

Basic configuration

The JUnit report displays the failed tests directly in the "Tests" tab of the MR. The principle is the same in the four languages below: ask the test runner to write an XML file, then declare it in artifacts:reports:junit.

Jest (Node.js)

test:
  image: node:20@sha256:8f693eaa7e0a8e71560c9a82b55fd54c2ae920a2ba5d2cde28bac7d1c01c9ba5
  script:
    - npm ci
    - npm test -- --reporters=jest-junit
  artifacts:
    when: always  # Important: even if the tests fail
    reports:
      junit: junit.xml

Install the reporter:

npm install --save-dev jest-junit

And configure jest.config.js:

module.exports = {
  reporters: [
    'default',
    ['jest-junit', { outputDirectory: '.', outputName: 'junit.xml' }]
  ]
};

pytest (Python)

test:
  image: python:3.11@sha256:c7220863385ee39fb6d822da81f4469d0cd33ff893d92ce94105e5c3f4b95fe2
  script:
    - pip install pytest pytest-cov
    - pytest --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml

Go

test:
  image: golang:1.21@sha256:4746d26432a9117a5f58e95cb9f954ddf0de128e9d5816886514199316e4a2fb
  script:
    - go install github.com/jstemmer/go-junit-report/v2@latest
    - go test -v ./... 2>&1 | go-junit-report -set-exit-code > report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml

Maven (Java)

test:
  image: maven:3.9-eclipse-temurin-17@sha256:42a3ac393abbc64dae6c96703e5ead2e921fe9103371ac58b5b404b0d6a26502
  script:
    - mvn test
  artifacts:
    when: always
    reports:
      junit: target/surefire-reports/*.xml

when: always

Without when: always, artifacts are not uploaded if the job fails. You will then not see the failed tests in the MR, which is the opposite of what you want!

Result in GitLab

The report does not appear in the job but in the merge request, once the pipeline has finished. GitLab compares the file with the latest report of the target branch, hence the notion of new failure. In the MR, you will see:

• A Tests tab with the number of passed/failed tests
• The list of failed tests with the error message
• A comparison with the target branch (new failures)

Code coverage

GitLab handles coverage through two independent mechanisms, often confused. The coverage: keyword extracts a global percentage from the job output, the one that feeds the badge and the value displayed on the MR. The coverage_report: report does something else: it colors the covered lines in the diff. The two add up, and most teams need both.

Display the global percentage

GitLab can extract the coverage % from the logs with a regex: it is the text printed by the test tool that is analyzed, not a file. The command must therefore really produce this summary in the job output, otherwise the value stays empty.

test:
  script:
    - npm test -- --coverage
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'

The regex is written between /.../ and, in YAML, between single quotes so that the \ and the | stay literal. It must contain a capture group (the parentheses) around the number to keep: the value of that group is what GitLab displays. For Jest, it skips the name column on the All files line of the text table, then captures the first number, the % Stmts column. Examples per framework:

Framework	Regex
Jest	/All files[^|]*|[^|]*\s+([\d\.]+)/
pytest-cov	/TOTAL\s+\d+\s+\d+\s+(\d+)%/
Go	/coverage:\s(\d+\.\d+)%/
JaCoCo	/Total.*?(\d+%)/

Swipe to see more

Display line-by-line coverage (Cobertura)

To see which lines are covered in the MR diff, you need a Cobertura report, an XML that lists the covered lines file by file. The paths it contains must match those of the repository, otherwise GitLab links no line to the diff:

Jest

test:
  script:
    - npm test -- --coverage --coverageReporters=cobertura
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

pytest

test:
  script:
    - pytest --cov=app --cov-report=xml
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Go

test:
  script:
    - go test -coverprofile=coverage.out ./...
    - go install github.com/boumenot/gocover-cobertura@latest
    - gocover-cobertura < coverage.out > coverage.xml
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml

Combine both

A single test run can feed both mechanisms: --coverageReporters=text prints the summary that the regex captures, cobertura writes the diff XML. Forgetting the text reporter is the classic mistake, coverage then stays at zero while the diff is correctly colored.

test:
  script:
    - npm test -- --coverage --coverageReporters=text --coverageReporters=cobertura
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

Code quality reports

The codequality report brings back into the MR the problems introduced by the modified lines: debt, complexity, lint rules. That is what makes the remark useful: nobody fixes the 4,000 warnings of an existing repository, but everybody fixes the three they have just added. GitLab imposes no tool, only an exchange format.

Code Climate format

GitLab displays quality issues in the MR if you use the Code Climate format, a JSON that describes each issue with its severity, its file and its line. Most linters can produce it through a dedicated formatter.

code_quality:
  image: node:20@sha256:8f693eaa7e0a8e71560c9a82b55fd54c2ae920a2ba5d2cde28bac7d1c01c9ba5
  script:
    - npm ci
    - npx eslint --format gitlab src/ > gl-code-quality-report.json || true
  artifacts:
    reports:
      codequality: gl-code-quality-report.json

|| true

|| true prevents the job from failing if ESLint finds errors. The report will still be generated and displayed.

ESLint with the GitLab formatter

The formatter maintained by GitLab converts the ESLint output to the expected format, with no intermediate script. The -o option writes the file directly, which avoids the shell redirection and the risk of capturing a stray log line that would invalidate the JSON.

npm install --save-dev @gitlab-formatters/eslint-formatter

code_quality:
  script:
    - npx eslint -f @gitlab-formatters/eslint-formatter -o gl-code-quality-report.json src/
  artifacts:
    reports:
      codequality: gl-code-quality-report.json

Use the GitLab Code Quality template

GitLab provides an official template: it runs a generic scanner in a container, detects the languages of the repository and produces the report without you having to configure a linter. Handy to get started, but it requires the Docker service in your runners and is less precise than a linter tuned for your project.

include:
  - template: Code-Quality.gitlab-ci.yml

code_quality:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Security reports

The security scanners provided by GitLab are enabled by a simple include: of a template: each one adds a job that produces a report in the in-house JSON format. These jobs run whatever the edition; what changes with the edition is the display of the results in the merge request and the vulnerability dashboard.

GitLab Ultimate

SAST, DAST and dependency scanning reports displayed in the MR require GitLab Ultimate. However, you can generate the reports and consult them as artifacts with all editions.

SAST (Static Application Security Testing)

SAST analyzes the source code looking for dangerous patterns, SQL injection or uncontrolled deserialization for example. The template automatically selects the analyzers matching the languages detected in the repository, there is nothing to declare.

include:
  - template: Security/SAST.gitlab-ci.yml

sast:
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"

Secret detection

This scanner looks for tokens and keys in the versioned files and, on an MR, in the added commits. A secret that is found must be considered compromised and revoked: removing it from the code is not enough, it stays in the Git history.

include:
  - template: Security/Secret-Detection.gitlab-ci.yml

secret_detection:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Dependency scanning

The dependency scanner reads the lock files (package-lock.json, poetry.lock, go.sum) and checks the installed versions against known vulnerabilities. It therefore finds nothing if the lockfile is not versioned.

include:
  - template: Security/Dependency-Scanning.gitlab-ci.yml

Dotenv report (variables)

The dotenv report lets you pass variables from one job to another: each KEY=value line of the file becomes an environment variable in the following jobs. It is the clean answer to the classic need to pass on a version number or the URL of a deployment, where an export in the script: disappears with the job's container.

build:
  stage: build
  script:
    - VERSION=$(cat version.txt)
    - echo "VERSION=$VERSION" >> build.env
  artifacts:
    reports:
      dotenv: build.env

deploy:
  stage: deploy
  needs: [build]
  script:
    - echo "Deploying version $VERSION"

Propagation

The variables of the dotenv report are automatically available in the jobs that have a needs: on the source job.

Best practices

Four settings make the difference between reports that really help and reports nobody looks at. They all concern the moment at which the report is produced and its effect on the merge request.

1. Always use when: always for tests

By default, GitLab uploads artifacts only if the job succeeds. On a test job, that is exactly the opposite of what you need: the report disappears at the very moment you need it.

test:
  script: npm test
  artifacts:
    when: always  # Otherwise no report if the tests fail
    reports:
      junit: junit.xml

2. Combine several reports

A single job can declare several reports: no need to duplicate the test run to produce the JUnit and the coverage separately. One run, several output formats, and the MR displays everything.

test:
  script:
    - npm test -- --coverage --reporters=jest-junit
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    when: always
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

3. Run on MRs

Reports are displayed in a merge request only if the pipeline was actually triggered for that MR. The first rule ensures this trigger, the second keeps a pipeline on main to have a comparison baseline.

test:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"

4. Do not block on quality (at first)

Plugging in a linter in blocking mode on an existing project makes the first MR that comes along fail, and the team disables the job within the week. allow_failure: true makes the report visible without blocking, while you pay down the debt before tightening the rule.

code_quality:
  script: npm run lint -- -f gitlab
  allow_failure: true  # Informational, does not block the MR
  artifacts:
    reports:
      codequality: gl-code-quality-report.json

Common errors

The common point of these five failures: the job goes green, no error is raised, and yet the MR stays empty. GitLab does not flag an unreadable report, it ignores it. When a report does not appear, first check the format produced and the actual presence of the file in the job's artifacts.

Error	Cause	Solution
Report not displayed	Incorrect format	Check the format (JUnit XML, Cobertura)
Failed tests invisible	when: always missing	Add when: always to the artifacts
Coverage 0%	Regex does not match	Test the regex against the logs
Code quality empty	Not Code Climate format	Use a GitLab formatter
"Test summary" missing	Invalid XML file	Validate the generated XML

Swipe to see more

Hands-on lab

The lab "Make the pipeline say what it covers" is played on a GitLab CE forge of your own, set up on a virtual machine with its runner and its registry, rather than on gitlab.com: no minute quota, no identity verification, and a project you can break as often as you need. dsoxlab run prepares the starting project there.

The tests run and say nothing about how much code they go through. You print the coverage, let GitLab fish it out of the log, and publish a report that shows the uncovered lines in a merge request. Grading is taken on what reached GitLab, not on the file. Since a regular expression that matches nothing raises no error, check the figure afterwards on the pipeline page or through the API: that is the only way to know the fishing worked.

The lab starts with dsoxlab run gl-10-rapports-de-qualite, after dsoxlab provision prepares the virtual machine, its runner and its registry. Installing DSOXLab and its prerequisites are described in the DSOXLab documentation. The sections above explain each setting: keep them at hand during the lab.

Guided lab Practice

Learn how to do it

Make the pipeline say what it covers

$ dsoxlab run gl-10-rapports-de-qualite

Environment: VM·Duration: ~30 min

How it is checked: the state of the system is checked after a reboot, not the command you typed.

First time? What DSOXLab is and how to install it

DSOXLab is the command line that prepares a disposable environment (an isolated shell or a VM), runs the lab scenario, then checks the real state of the system and computes a score. Nothing is installed or broken on your workstation.

It needs Python 3.11 or later, uv or pipx, and git. The installation guide details these prerequisites; on Windows, or on a Mac without Python, use the appliance instead.

This lab starts virtual machines: it also needs ansible-playbook, a hypervisor (KVM or Incus), Terraform and hardware virtualization (/dev/kvm), which inside a VirtualBox VM means nested virtualization. The first time, dsoxlab start gl-10-rapports-de-qualite checks all of it before creating the machines. Prepare your machine for vm labs.

Install the CLI and add the catalogue, once:

uv tool install dsoxlab   # or: pipx install dsoxlab
dsoxlab catalog add https://github.com/stephrobert/gitlab-ci-dsoxlab-training

Before you run the lab, check:

• DSOXLab installed
• catalogue available
• compatible environment (isolated shell or VM)

To check, nothing is probed from this page: run dsoxlab doctor. Read the diagnosis

Run this lab, then validate your work:

dsoxlab run gl-10-rapports-de-qualite
dsoxlab check gl-10-rapports-de-qualite

See every lab: dsoxlab list-labs. Track your progress: dsoxlab progress. The full dsoxlab guide covers scoring and the pitfalls.

Key takeaways

Each type of report answers a different question and requires its own format. The table below serves as a memo when writing a .gitlab-ci.yml. The right-hand column keeps the detail that makes the display fail when it is forgotten.

Report	Usage	Key point
artifacts:reports	File interpreted by GitLab and displayed in the MR	artifacts:paths only stores
junit	Failed tests in MR	when: always mandatory
coverage:	Global % in the badge	Regex on the logs
coverage_report:	Line-by-line diff	Cobertura format
codequality:	ESLint/SonarQube issues	Code Climate format
sast:	Code vulnerabilities	GitLab Ultimate for MR
dotenv:	Variables between jobs	Propagated via needs:

Swipe to see more

Knowledge check

Test your knowledge of GitLab quality reports.

Knowledge check

Check what you have learned with this interactive quiz

10 questions

5 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →