GitLab CI services and advanced cache

Do your integration tests need a real database, not a mock? GitLab CI/CD runs services (PostgreSQL, Redis, Elasticsearch...) as containers next to your job. Your code connects to them as if they were local. Combined with a well-configured advanced cache, your pipelines are both complete and fast.

The Artifacts and cache lesson covered the basics. This guide goes deeper into advanced cases: Docker services, cache key strategies, policies and distributed cache.

What you will learn

• Run services (PostgreSQL, Redis, MongoDB) in a CI job
• Configure the connection between the job and the service (hostname, variables)
• Master cache policies: pull-push, pull, push
• Design robust cache keys (lock file, combinations)
• Manage the distributed cache (S3) for autoscaled runners
• Diagnose service and cache problems

Prerequisites

This lesson assumes you can write a pipeline with cache: and variables:, and that you understand how a runner executes a job. The examples target a runner with the Docker executor: the job and each service run as containers. The runner's config.toml settings (service wait timeout, S3 distributed cache) need access to the runner machine, or an administrator who applies them for you; services and cache keys only need edits to the .gitlab-ci.yml.

• Writing a .gitlab-ci.yml file: stages, jobs, image: and script:
• GitLab runners: executors, config.toml and running a job in a container
• Variables and secrets: declaring a variable and reading it in a job
• Artifacts and cache: the cache basics, which this lesson goes deeper into

In which context?

Your unit tests pass, but your integration tests fail because they need a database. You could mock it, but the tests would not be realistic. You need a real PostgreSQL (or Redis, or MongoDB) in your pipeline.

At the same time, your cache is unpredictable: sometimes it is there, sometimes it is empty. Builds take 3 minutes with the cache, 12 without. You want to understand why and make it reliable.

• Your integration tests need an SQL or NoSQL database
• You are testing a microservice that depends on Redis for sessions
• Your app has a healthcheck that verifies the database connection
• Your cache disappears when the runner changes
• You have autoscaled runners and the local cache does not survive

CI services

How do services work?

When you declare services: in a job, GitLab Runner:

1. Starts the service containers before the job
2. Waits for the services to be ready (healthcheck)
3. Creates a shared network between the job and the services
4. Makes each service reachable through a hostname derived from the image name

Docker-in-Docker not needed

CI services are sibling containers, not containers inside a container. The Docker runner creates them at the same level as your job. There is no need for docker:dind to use a PostgreSQL in CI.

PostgreSQL in a job

The most common example: integration tests with PostgreSQL.

.gitlab-ci.yml

test:integration:
  stage: test
  image: node:24-alpine@sha256:a0b9bf06e4e6193cf7a0f58816cc935ff8c2a908f81e6f1a95432d679c54fbfd
  services:
    - name: postgres:16-alpine@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685
      alias: db
      variables:
        POSTGRES_DB: testdb
        POSTGRES_USER: testuser
        POSTGRES_PASSWORD: testpass
  variables:
    DATABASE_URL: "postgresql://testuser:testpass@db:5432/testdb"
  script:
    - npm ci
    - npm run test:integration

Key to understanding: the service is reachable through the hostname db (defined by alias). Without an alias, the hostname would be postgres: the image name without its tag or digest (for a namespaced image such as registry.example.com/team/postgres, the / become -).

Redis for sessions

Redis needs no environment variable to start, unlike PostgreSQL: the official image accepts connections without authentication from launch. The only thing to remember from this example is therefore the construction of the connection URL, which reuses the service alias and the port exposed by default, 6379. If you want a password, it must be passed as a command to the service, not as a variable.

test:integration:
  stage: test
  image: python:3.12@sha256:7ad6d21a25a94b2c00e685e82c2fd298de814353d9ee0e3f7f2cd4fca063df60
  services:
    - name: redis:7-alpine@sha256:ff02b58f971e7d7d156a1267e283fcbbeee91773b6aa36c49dac28ecfe28eadf
      alias: cache
  variables:
    REDIS_URL: "redis://cache:6379"
  script:
    - pip install -r requirements.txt
    - pytest tests/integration/

Several simultaneous services

Does your application depend on PostgreSQL and Redis? Declare both:

test:integration:
  stage: test
  image: node:24-alpine@sha256:a0b9bf06e4e6193cf7a0f58816cc935ff8c2a908f81e6f1a95432d679c54fbfd
  services:
    - name: postgres:16-alpine@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685
      alias: db
      variables:
        POSTGRES_DB: app_test
        POSTGRES_USER: app
        POSTGRES_PASSWORD: secret
    - name: redis:7-alpine@sha256:ff02b58f971e7d7d156a1267e283fcbbeee91773b6aa36c49dac28ecfe28eadf
      alias: cache
    - name: elasticsearch:8.12.0@sha256:2529e4b8062e4b9f0a31bea3ee797a58dae4cfe2d2b5879b4a1b03250c7ad31c
      alias: search
      variables:
        discovery.type: single-node
        xpack.security.enabled: "false"
  variables:
    DATABASE_URL: "postgresql://app:secret@db:5432/app_test"
    REDIS_URL: "redis://cache:6379"
    ELASTICSEARCH_URL: "http://search:9200"
  script:
    - npm ci
    - npm run test:integration

Runner memory

Each service is a container. Three services + the job = 4 simultaneous containers. Check that your runners have enough memory. A runner with 2 GB of RAM will not support PostgreSQL + Elasticsearch + Redis + your app.

How do you wait for a service to be ready?

Services do not start instantly. PostgreSQL needs a few seconds to be ready. Two approaches:

Wait script

test:integration:
  services:
    - name: postgres:16-alpine@sha256:cf78e76683b9ca8c5733cbbdce6c9262b45b6767934dd0a95e671f9a0fc20685
      alias: db
  before_script:
    - apt-get update && apt-get install -y postgresql-client
    - |
      for i in $(seq 1 30); do
        pg_isready -h db -p 5432 && break
        echo "Waiting for PostgreSQL... ($i/30)"
        sleep 1
      done
    - npm ci
  script:
    - npm run test:integration

Built-in healthcheck

GitLab Runner does not read the image's HEALTHCHECK directive. It reads the ports exposed by the service container, then starts an auxiliary container that waits for those ports to accept a connection. This is a TCP check, not an application check: PostgreSQL's port may answer while the database does not yet accept requests.

Two practical consequences. An image with no exposed port will never be considered ready. And if the delay is exceeded, the runner writes WARNING: Service ... probably didn't start properly and then continues the job anyway, which will fail later on a connection error. That is why the previous tab, with pg_isready, remains the reliable method for a service with a long initialization.

To increase the wait delay, whose default value is 30 seconds, configure in the runner:

config.toml

[[runners]]
  [runners.docker]
    wait_for_services_timeout = 60

Advanced cache

Reminder: cache vs artifacts

The line that decides everything in this table is the lifetime. The cache is a best effort mechanism: GitLab never guarantees that it will be there at the next job, and a pipeline that fails when the cache is empty is a broken pipeline, not a slow one. Artifacts, on the other hand, are kept for the time declared by expire_in and are the only reliable way to pass a result from one job to another. If you hesitate between the two, ask yourself this question: does the next job fail without this file? If so, it is an artifact.

	Cache	Artifacts
Purpose	Speed up following builds	Pass results between jobs
Lifetime	Best effort (may disappear)	Guaranteed during expire_in
Storage	Local to the runner (or distributed)	Stored by GitLab (object storage)
Use case	node_modules/, .pip/, vendor/	dist/, JUnit reports, binaries

Swipe to see more

→ Details in Artifacts and cache

Which cache key strategy should you choose?

The cache key determines when the cache is reused or invalidated.

By lock file

cache:
  key:
    files:
      - package-lock.json
  paths:
    - node_modules/

The best choice in most cases. The cache is invalidated only when the dependencies change. Two branches with the same package-lock.json share the cache.

This is the move the lab of the DAG and parallelism lesson makes you fix: its starting pipeline keys the cache on the commit SHA, so the cache is never reused from one commit to the next.

By branch

cache:
  key: $CI_COMMIT_REF_SLUG
  paths:
    - node_modules/

Each branch has its own cache. Simple but wastes space if the dependencies are identical between branches.

Combined

cache:
  key:
    prefix: $CI_COMMIT_REF_SLUG
    files:
      - package-lock.json
  paths:
    - node_modules/

Cache by branch and by lock file. The cache is invalidated if the branch or the dependencies change. Useful if different branches test different dependency versions.

Fallback keys

When the exact cache does not exist, GitLab can look for a fallback cache. The point to understand before writing a single line: fallback_keys contains literal key names, not patterns. A fallback key therefore only brings back a cache if another pipeline actually pushed one under that exact name. This is why the example below does not use key.files: a key computed by hashing produces an unpredictable name, on which no fallback key can ever land.

cache:
  key: "npm-$CI_COMMIT_REF_SLUG"
  fallback_keys:
    - "npm-$CI_DEFAULT_BRANCH"
    - "npm-default"
  paths:
    - node_modules/

The search order, from a feature/paiement branch:

1. npm-feature-paiement, the exact key of the current branch
2. npm-main, written by the last pipeline of the default branch
3. npm-default, a safety net that a maintenance job must feed

The second entry is the one that makes the mechanism useful: a freshly created branch retrieves the node_modules/ of the default branch instead of starting from an empty cache.

Policies: controlling who writes and who reads

The policy answers a question of cost, not of correctness: each cache write compresses a directory and sends it to storage, which takes time even when nothing has changed. On a pipeline where five jobs declare the same node_modules/, leaving the default value pull-push everywhere amounts to compressing and uploading the same content five times. Switching the consuming jobs to pull removes these useless writes and eliminates along the way the concurrent overwrites between parallel jobs.

Policy	Behavior	Usage
pull-push	Reads the cache at the start, writes at the end (default)	The job that feeds the cache (often the first)
pull	Reads the cache, does not write	The following jobs (avoids concurrent writes)
push	Does not use the existing cache, writes at the end	Regenerating a corrupted cache

Swipe to see more

Example: one job feeds, the others consume

stages: [lint, test, build]

lint:
  stage: lint
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
    policy: pull-push  # First job → feeds the cache
  script:
    - npm ci
    - npm run lint

test:
  stage: test
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
    policy: pull  # Following job → read-only
  script:
    - npm test

build:
  stage: build
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
    policy: pull  # Read-only too
  script:
    - npm run build

Notice that test and build repeat the key and the paths of the first job, and not only policy: pull. This is deliberate: a cache: block declared in a job replaces the one from default: entirely, with no merging. A job that wrote only policy: pull would have no key and no path, and would therefore restore nothing, without any error message. To avoid the repetition without losing this rule, put the shared block in a hidden job (.node-cache) and reuse it with extends:, overriding only the policy.

Multiple caches

A job can declare several independent caches:

test:python:
  cache:
    - key:
        files:
          - requirements.txt
      paths:
        - .pip/
    - key:
        files:
          - package-lock.json
      paths:
        - node_modules/
  script:
    - pip install -r requirements.txt --cache-dir .pip
    - npm ci
    - npm test

Distributed cache (S3)

By default, the cache is stored locally on the runner. If your runners are autoscaled (ephemeral instances), the cache disappears with the instance. The solution: a distributed cache on S3 (or S3-compatible storage).

Configuration in the runner:

config.toml

[[runners]]
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "gitlab-ci-cache"
      BucketLocation = "eu-west-1"
      AccessKey = "AKIAIOSFODNN7EXAMPLE"
      SecretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

The last two lines are the example credentials published by AWS, they give access to nothing. In production, do not copy this pattern: config.toml is a plain-text file on the runner, readable by anyone who gets a shell on it. On EC2 or EKS, remove AccessKey and SecretKey and set AuthenticationType = "iam" so that the runner uses the IAM role of the instance. On S3-compatible storage with no role available, inject the keys through environment variables of the runner service and restrict the bucket policy to the strict minimum.

S3-compatible alternatives

The distributed cache works with any S3-compatible storage: MinIO (self-hosted), Google Cloud Storage (with S3 interoperability), DigitalOcean Spaces. You are not tied to AWS.

How do you clear the cache?

The cache can become large. Important point: there is no REST API endpoint to clear a runner's cache. The GitLab documentation only documents two ways, and the interface button is the most direct.

In the interface: Build > Pipelines, then Clear runner caches (button at the top right). On the next commit, jobs start again on a fresh cache location.

The second way is to change the cache key (for example by adding a prefix: v2 to it), which moves the jobs to a new entry without deleting the old one.

Do not confuse this with deleting artifacts, which falls under a completely different API endpoint: DELETE /projects/:id/artifacts deletes the eligible artifacts of a project, and DELETE /projects/:id/jobs/:job_id/artifacts those of a specific job. Neither touches the cache.

For a regular refresh, add a scheduled job that rebuilds the cache from scratch under the same key as the consuming jobs. Its push policy ignores the existing cache and overwrites it at the end of the job:

stages: [maintenance]

clean:cache:
  stage: maintenance
  cache:
    key:
      files:
        - package-lock.json
    policy: push
    paths:
      - node_modules/
  script:
    - rm -rf node_modules/
    - npm ci
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"

Troubleshooting

Before going through this table, open the job logs and go back to the very beginning: the runner writes a collapsed section per service there, with the result of the port check. A WARNING: Service ... probably didn't start properly message points you to the service rows (connection refused, memory, MySQL), whereas a connection error without this message rather points to an alias or variables problem. The cache rows (empty cache, inconsistent cache, never invalidated, pull job that reads nothing) share one visible symptom: a build duration that varies for no apparent reason.

Symptom	Likely cause	Solution
Service connection refused	The service is not ready	Add a wait script or increase wait_for_services_timeout
Service hostname not found	Wrong alias or image syntax	Check alias: and the hostname used in the connection
Empty cache on a new branch	No fallback configured	Add fallback_keys pointing to the default branch
Inconsistent cache between jobs	Concurrent write	Use policy: pull on all jobs except the first
Cache never invalidated	Static key (key: "npm-cache")	Use key.files with a lock file
pull job restores nothing	Its cache: block replaces the default: one and lacks key or paths	Repeat key and paths in the job (see the policies section)
Out of memory with services	Too many containers	Reduce the services or increase the runner resources
MySQL service Access denied	Env variables not passed to the service	Declare the variables under services[].variables:

Swipe to see more

Key takeaways

• CI services run sibling containers (PostgreSQL, Redis...) reachable by hostname
• The hostname is the service alias or the simplified image name
• Each service has its own variables (POSTGRES_DB, REDIS_URL...)
• The cache key based on a lock file is the most reliable strategy
• Use policy: pull-push on the first job and pull on the following ones, repeating key and paths in each: a job's cache: replaces the default: one entirely
• Fallback keys avoid cache misses on new branches
• The distributed cache (S3/MinIO) is essential for autoscaled runners
• Cache = best effort (accelerator), artifacts = guaranteed (results)

Test your knowledge

Ten questions to check that the two mechanisms of the guide are mastered: the network naming of services and the key/policy pair of the cache. The pass mark is set at 70%. If you struggle on the policy questions, reread the three-job example above, it is the point that costs the most pipeline time in practice.

Knowledge check

Check what you have learned with this interactive quiz

10 questions

5 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →