CI/CD best practices: the invariants of a reliable pipeline

A reliable pipeline rests on six invariants: build only once, never modify an artifact, produce the same result on every run, answer within minutes, fail fast and clearly, and expose its health. Two teams can use the same CI tool and get opposite results: for one, a failure is located within seconds; for the other, runs diverge and nobody can reproduce the breakdown. The gap does not come from the tool, GitHub Actions, GitLab CI or Jenkins, but from the principles applied. This lesson presents them for a reader who already maintains a pipeline.

What you will learn

• Apply artifact promotion (build once, deploy many)
• Guarantee artifact immutability and run idempotence
• Speed up feedback through parallelism, cache and fail fast
• Separate building from deploying, configuration included
• Measure a pipeline's health and deal with flaky tests

Why do these practices matter?

A CI/CD pipeline is not a script you write once and forget: it is a critical system that validates every change, automates deployments, holds production secrets and drives the productivity of the whole team. When it is badly designed, you wait hours for results, bugs slip through because "the tests were green", and developers end up bypassing it.

A good pipeline, on the contrary, gives feedback within minutes, the certainty that what is tested is what is deployed, predictable deployments, and a team that trusts its tooling. The following sections detail the practices that produce this result, in the order in which they protect the deliverable.

Build once, deploy many

The artifact (image, binary, package) is built only once, then deployed as is to every environment. The opposite pattern, rebuilding for each environment, is common and risky: between the staging build and the production one, a dependency, the base image or a build tool may have changed, and the production artifact is no longer the one that was tested.

# ❌ Rebuilding for each environment: to avoid
staging:
  script:
    - npm install          # versions may differ on the next build
    - npm run build
    - docker build -t app:staging .
    - docker push registry/app:staging

production:
  script:
    - npm install          # dependencies resolved again
    - npm run build        # potentially different result
    - docker build -t app:production .
    - docker push registry/app:production

Artifact promotion builds once, with a unique tag per commit, then deploys that same artifact everywhere. Only the configuration changes from one environment to the next: variables, secrets, database addresses. The artifact itself never changes.

build:
  script:
    - npm ci
    - npm run build
    - docker build -t registry/app:$COMMIT_SHA .   # unique tag per commit
    - docker push registry/app:$COMMIT_SHA

staging:
  script:
    - helm upgrade app ./chart --set image.tag=$COMMIT_SHA -f values-staging.yaml

production:
  script:
    - helm upgrade app ./chart --set image.tag=$COMMIT_SHA -f values-production.yaml

The table compares the two approaches on the criteria that matter during an incident. Read the rollback row first: with a rebuild, going back means replaying a build that may fail at the worst moment; with promotion, you just redeploy a tag already in the registry.

Aspect	Rebuild per environment	Artifact promotion
Reproducibility	Uncertain	Guaranteed
Confidence	"It worked in staging..."	"It's the same artifact"
Deployment time	Long (full rebuild)	Fast (deployment only)
Rollback	Rebuild the old version	Redeploy the old tag

Swipe to see more

Promotion expects three things from the application. It separates code from configuration: no API address hard-coded, but a variable read at startup. It accepts its configuration at runtime, through environment variables, mounted files or injected secrets. And its artifacts are tagged uniquely, by the commit SHA, rather than app:staging, which does not say which code it contains.

Immutability: what is created never changes

An artifact, once created, must never be modified: if a change is needed, a new artifact is created. Take an app:latest image deployed on Monday; on Tuesday, a colleague rebuilds app:latest with a fix; on Wednesday, another service restarts and pulls that new image; on Thursday, it no longer works. The tag is the same, the content changed: that is the absence of immutability.

What breaks immutability, and what preserves it

The faulty practices share one thing: they let an artifact's content evolve while its identifier stays frozen. The most frequent case is mutable tags such as latest: two pulls a week apart may bring back different images, and nothing links what runs any more to the code that produced it.

Faulty practice	Problem
Mutable tags (latest, stable, main)	The content changes, the name stays
Modifying an artifact after creation	No way to know what really runs
Scripts that change production by hand	State differs from what is versioned
Deployment without an explicit tag	Nobody knows which version is in production

Swipe to see more

On the other hand, four mechanisms guarantee that an identifier always points to the same content: tagging by commit SHA, frozen semantic versions, infrastructure described as versioned code, and a registry that forbids overwriting an existing tag. The rule boils down to one question: if I replay this pipeline in six months, will I get the same artifact?

# ❌ Mutable tag: its content may change
docker build -t app:latest .
docker push app:latest

# Tag by commit SHA: the source of truth
docker build -t app:$CI_COMMIT_SHA .
docker push app:$CI_COMMIT_SHA

# A version tag can be added for readability, the SHA stays the reference
docker tag app:$CI_COMMIT_SHA app:v1.2.3
docker push app:v1.2.3

Idempotence: same commit, same result

Running the same pipeline on the same commit must produce exactly the same result, on the first run as on the tenth. Without that guarantee, debugging becomes impossible ("it worked yesterday"), rollback becomes uncertain, since rebuilding the old version does not give back the same artifact, and tests pass or fail at random.

The sources of non-determinism

Each cause in the table introduces a source of non-determinism: something that varies from one run to the next while the commit has not moved. The first one is the most common: an npm install without a lock file may bring back a different version without any of your files having changed.

Cause	Example	Problem
Unpinned dependencies	npm install without a lock file	A dependency changes between two builds
Timestamp in the artifact	Build date written into the binary	The artifact changes on every build
External data	Tests that call a real API	The API may answer differently
Modified global state	Script that changes a global variable	The second run sees a different state
Non-deterministic order	Badly isolated parallel tests	Tests influence each other

Swipe to see more

Pin every dependency

The first countermeasure is to install dependencies from a versioned lock file. npm ci installs exactly what package-lock.json describes. On the Python side, a requirements.txt produced by pip-compile --generate-hashes freezes each dependency, transitive ones included, with its digest, and pip install --require-hashes refuses any package that does not match.

# ❌ Non-deterministic: resolution may change tomorrow
npm install
pip install requests

# Deterministic: versions and digests frozen by the lock files
npm ci
pip install --require-hashes -r requirements.txt

Each ecosystem has its lock file, to version with the code. The table lists the main ones; a single file missing from the repository is enough to make the build non-reproducible.

Ecosystem	Lock file
Node.js (npm)	package-lock.json
Node.js (yarn)	yarn.lock
Python (pip)	requirements.txt pinned with digests
Python (poetry)	poetry.lock
Go	go.sum
Rust	Cargo.lock

Swipe to see more

Rule out time, shared state and the network

Three other causes are dealt with in the code. A build date written into the artifact changes its content on every build: write the commit SHA instead. A test that assumes a global state depends on the tests run before it: each test creates its own context. And a test that calls a real API depends on its availability: it is replaced by a mock.

# ❌ The date changes on every build
RUN echo "Built on $(date)" > /app/version.txt

# Deterministic information: the commit that produced the image
ARG COMMIT_SHA
RUN echo "Commit: $COMMIT_SHA" > /app/version.txt

# ❌ Depends on the other tests and on the network
def test_user_count():
    assert User.count() == 5

# Clean context and mocked external dependency
def test_user_count():
    with fresh_database():
        create_users(5)
        assert User.count() == 5

def test_weather(mock_weather_api):
    mock_weather_api.return_value = {"temp": 20}
    assert get_weather("paris")["temp"] == 20

Fast feedback: knowing within minutes whether the code is valid

A developer must know within a few minutes whether their code is valid. A test's value lies not only in what it detects, but in the delay after which it reports it: a failure reported after forty minutes arrives when the developer has switched context, and the fix takes all the longer.

The table gives realistic orders of magnitude per stage. Beyond half an hour in total, developers start bypassing the pipeline ("I'll merge and see"); the overall target to aim for stays under fifteen minutes.

Stage	Target	Why this duration
Lint and formatting	< 1 min	Syntax checks
Unit tests	< 3 min	Isolated tests, no external dependency
Build	< 5 min	Compilation, packaging
Integration tests	< 10 min	Database, services
End-to-end tests	< 15 min	Browser, complete scenarios

Swipe to see more

How do you speed up a pipeline?

Four levers reduce the duration without sacrificing coverage. The first is parallelism: lint, tests and security scan, which are independent, run at the same time rather than one after the other. The second is fail fast: if the lint fails in three seconds, there is no point running tests that take thirty.

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

The third lever is the dependency cache, which avoids downloading everything again: an npm ci that took three minutes drops to a few seconds when its packages are already there. The fourth is the test pyramid: many fast unit tests, few slow end-to-end tests, kept for the essential journeys.

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Separate building from deploying

The build creates a generic artifact, the deployment installs it with an environment-specific configuration, and the two do not mix. A build that knows its destination, for example an API address injected at compile time, produces one artifact per environment: it becomes impossible to promote the one that was tested.

# ❌ Coupling: the build knows the destination
build-staging:
  script:
    - API_URL=https://api.staging.internal npm run build
    - docker build -t app:staging .

build-production:
  script:
    - API_URL=https://api.internal npm run build
    - docker build -t app:production .

Separation fixes this coupling: the build receives no environment configuration, and each deployment injects its own at startup. The same artifact thus goes to staging then to production, and a new environment only needs a new values file, not a new build.

build:
  script:
    - npm run build                      # no environment configuration
    - docker build -t registry/app:$COMMIT_SHA .
    - docker push registry/app:$COMMIT_SHA

deploy-staging:
  needs: [build]
  script:
    - helm upgrade app ./chart --set image.tag=$COMMIT_SHA -f values-staging.yaml

deploy-production:
  needs: [build]
  script:
    - helm upgrade app ./chart --set image.tag=$COMMIT_SHA -f values-production.yaml

The table sets each stage's role. The column that protects promotion is the last one: as long as the build injects no address or secret, and the deployment recompiles nothing, the validated artifact is indeed the one that goes to production. As soon as a stage oversteps its role, the guarantee "what is tested is what is deployed" falls.

Stage	Responsibility	What it does not do
Build	Compile, package, create the artifact	Inject addresses, secrets or environment configuration
Test	Validate that the artifact works	Modify the artifact
Deploy	Install the artifact and inject the configuration	Recompile, modify the code

Swipe to see more

Declarative pipeline: describe what, not how

The pipeline file describes what you want to obtain, and the platform handles orchestration. A big imperative script, made of nested if blocks that build, push and deploy depending on the branch, is hard to read, to maintain and to reuse, and the platform cannot parallelise anything in it. A declarative description, on the contrary, exposes a graph of jobs that the platform knows how to optimise.

The following example, on GitHub Actions, shows its shape. Permissions are emptied at workflow level then granted job by job, actions are pinned by commit SHA, and the production environment carries an approval rule configured in the repository. The connection to the Kubernetes cluster, specific to each provider, is omitted.

permissions: {}

jobs:
  build:
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      packages: write          # publish the image to the repository registry
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
        with:
          persist-credentials: false
      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020  # v7.0.0
        with:
          node-version: '24'
      - run: npm ci
      - run: npm run build
      - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f  # v4.6.0
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc  # v7.4.0
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

  deploy-staging:
    needs: build
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    permissions:
      contents: read
    environment: staging
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
        with:
          persist-credentials: false
      - uses: azure/k8s-deploy@bf1ec05e1d736ac4ac70bae889d3ec48c79e2213  # v7.0.1
        with:
          manifests: k8s/staging/
          images: ghcr.io/${{ github.repository }}:${{ github.sha }}

  deploy-production:
    needs: deploy-staging
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-24.04
    permissions:
      contents: read
    environment: production    # approval required by the environment rule
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
        with:
          persist-credentials: false
      - uses: azure/k8s-deploy@bf1ec05e1d736ac4ac70bae889d3ec48c79e2213  # v7.0.1
        with:
          manifests: k8s/production/
          images: ghcr.io/${{ github.repository }}:${{ github.sha }}

Five keywords form the vocabulary shared by most platforms, even if the syntax varies. Remember needs above all, which declares the dependencies between jobs: it builds the execution graph and tells the platform what it can run in parallel.

Concept	Meaning	Example
needs	"This job depends on that one"	needs: [build, test]
if	"Run only if..."	if: github.ref == 'refs/heads/main'
environment	"Deploy to this environment"	environment: production
matrix	"Run for each combination"	matrix: { node: ['22', '24'] }
uses	"Use this reusable action"	uses: actions/checkout@<commit SHA>

Swipe to see more

Fail fast, fail loud: failing quickly and clearly

When something goes wrong, the pipeline must stop immediately and report it clearly. A silent error lets the bug reach production; a late error arrives after ten other merges, and nobody knows which one broke things any more; a loud, immediate error is fixed within minutes.

Stop at the first problem

A pipeline script must never carry on with an invalid state. By default, a shell chains commands even when one of them fails, and a deployment can go out on a failed build. The set -euo pipefail line at the top of a script stops everything at the first failure.

#!/usr/bin/env bash
set -euo pipefail   # stop at the first failure, undefined variable or failed pipe

npm run build
npm test
./deploy.sh         # never reached if a previous step failed

Each of the three options closes a different leak. The table details them; -o pipefail is the least known and the most useful, because a pipe such as command | tee log.txt otherwise succeeds as soon as tee succeeds, even if the command failed.

Option	Effect
-e	Stops the script as soon as a command returns an error
-u	Stops the script if an undefined variable is used
-o pipefail	Treats a pipe as failed as soon as one of its commands fails

Swipe to see more

Notify clearly and make diagnosis easy

A useful failure notification arrives immediately, to the commit's author rather than to the whole team, and says what failed: "Test user_login_test: expected 200, got 401" rather than "Pipeline failed". It points directly to the faulty line. The developer must be able to understand the failure without digging through 500 lines of logs.

To get there, the pipeline structures its logs by stage, keeps the context (versions, variables minus secrets), publishes diagnostic artifacts such as test screenshots or reports, and says how to reproduce the failure locally, for example with make test-integration.

Observability: seeing what is going on without digging

The pipeline exposes its health proactively: you should not have to read the logs to know whether all is well. Two metrics deserve special attention: the success rate, which reveals a pipeline losing the team's trust, and flaky tests, which erode that trust quietly. The thresholds in the table are starting points, to adjust to your context.

Metric	What it reveals	Starting alert threshold
Total duration	Performance regression	> 15 min
Duration per job	Bottleneck	A job whose duration doubles
Success rate	Overall health	< 90% over 24 h
Flaky tests	Unreliable tests	The same test failing at random several times a week
Queue time	Lack of runners	> 5 min

Swipe to see more

Flaky tests, a silent poison

A flaky test fails now and then for no apparent reason, then passes when re-run. It is serious because the team gets used to re-running without reading, real failures drown in the noise, and wasted time piles up. They are spotted by tracking the tests that fail then pass on the same commit.

The treatment fits in three steps. Isolate the test so it no longer blocks the others, open a priority ticket, and fix the root cause, almost always a dependency on time, a shared state or an external resource. A test isolated and forgotten is a deleted test that does not say so.

Before changing a pipeline

Before merging a pipeline change, eight quick checks go back over the invariants of this lesson. They fit into a review of a few minutes and avoid sneaking in a mutable tag, an unpinned dependency or a per-environment build.

• Build once: the artifact is built only once.
• Immutability: tags are SHAs, never latest.
• Idempotence: lock files are versioned and used.
• Fast feedback: the full pipeline runs in under fifteen minutes.
• Fail fast: a failure stops the pipeline immediately.
• Notifications: alerts point to the cause, with its context.
• Separation: build and deployment are distinct jobs.
• Declarative: the logic lives in the structure, not in scripts.

Key points

1. Build once, deploy many: build the artifact once, deploy it everywhere; only the configuration changes.
2. Immutability: a created artifact never changes; tag by SHA, not by latest.
3. Idempotence: same commit, same result; pin dependencies, isolate tests, rule out timestamps.
4. Fast feedback: parallelise, cache, fail fast; aim for under fifteen minutes.
5. Separate build and deployment: the build is generic, the deployment injects the configuration.
6. Declarative pipeline: describe the what, let the platform orchestrate.
7. Fail fast, fail loud: set -euo pipefail, notifications that name the cause.
8. Observability: track duration, success rate and flaky tests, and alert on what is actionable.

Knowledge check

Check that the essentials of this lesson are in place. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →