CI/CD anti-patterns: recognising and fixing pipeline mistakes

A CI/CD anti-pattern is a solution that saves time today and costs time for months: a 45-minute monolith, "re-run it, it'll pass", copy-paste between projects, temporary exceptions turned permanent. This lesson describes eight common anti-patterns by their symptom, explains why each one is a problem, and gives its fix. It is for anyone who maintains an existing pipeline.

What you will learn

• Recognise an anti-pattern by its symptom, before it gets expensive
• Fix the monolith, non-determinism and copy-paste
• Apply artifact promotion and the test pyramid
• Frame exceptions with an expiry date
• Audit a pipeline with a quarterly checklist

What is an anti-pattern?

An anti-pattern is a solution that looks good at the time but creates problems in the long run. The gain is immediate and visible, the cost is deferred and diffuse: this asymmetry is what makes it a trap. You adopt it to ship faster today, you pay for it in maintenance time and incidents for months.

In the context of CI/CD pipelines (Continuous Integration / Continuous Delivery, the automated system that tests and deploys your code), anti-patterns are practices that:

• Slow down feedback to developers
• Make maintenance difficult
• Create bugs that are hard to reproduce
• Pile up technical debt (problems pushed back to later)

Why this guide?

The most problematic pipelines are not the ones that fail loudly. They are the ones that seem to work but pile up invisible debt: build time that keeps growing, flaky tests that get ignored, exceptions that stack up.

The goal is not to shame anyone, but to recognise familiar situations and identify ways to improve.

The monolithic pipeline

A monolithic pipeline chains every step of the cycle in a single definition, from lint to production deployment. Each run pays the cost of the whole, even when the change only concerns one step, and a failure on any link blocks everything else.

Symptom

A single pipeline does everything: code check (lint), tests, compilation (build), security analysis (scan), deployment to every environment. It takes 45 minutes. Nobody dares touch it.

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

Why it is a problem

The four rows of the table stem from one cause: the lack of splitting. Focus on the first one, slow feedback, because it is the one that hurts daily work the most. A developer who waits forty-five minutes to learn that a comma breaks the build ends up no longer waiting: they move on to something else, lose the context, and come back cold to their mistake. The other three rows are the structural consequences of the same monolith.

Problem	Concrete impact
Slow feedback	45 minutes to find out whether a typo breaks the build
Fragility	A change in one step can break all the others
No parallelism	Steps run one by one, even if they could run in parallel
Hard debugging	When it fails, you have to search 500 lines of configuration

Swipe to see more

How to fix it

Split into independent pipelines with clear responsibilities. Each pipeline has a single goal and can fail without blocking the others:

CI pipeline (5 min)           ← Checks that the code is correct
├── lint                      ← Checks the code style
├── unit tests                ← Tests functions individually
└── build                     ← Compiles the application

CD-staging pipeline (3 min)   ← Deploys for testing
├── deploy staging            ← Puts it online in the test environment
└── e2e tests                 ← Exercises the application through its interface

CD-prod pipeline (2 min)      ← Deploys for real users
├── deploy production
└── smoke tests               ← Checks that the essential functions work

Immediate benefit: if the lint fails (2 minutes), the developer knows right away instead of waiting 45 minutes.

The non-deterministic pipeline

A deterministic pipeline always gives the same result with the same inputs. A non-deterministic pipeline introduces randomness, and therefore uncertainty.

Symptom

The same pipeline on the same code gives different results. The tell-tale sentence: "Re-run it, it'll pass."

Run 1: ✓ success
Run 2: ✗ failure (test timeout)    ← Why now?
Run 3: ✓ success
Run 4: ✗ failure (dependency not found)
Run 5: ✓ success                   ← Nothing changed though...

Common causes

These four causes have one thing in common: they introduce an uncontrolled input into a process that is supposed to be reproducible. Unpinned dependencies and flaky tests account for most cases, and they are also the easiest to fix. First identify which one hits you by reading the failure message: a "module not found" points to dependencies, an intermittent "timeout" or "connection refused" to flaky tests or shared resources.

Cause	Simple explanation
Unpinned dependencies	npm install downloads the latest version, which may have changed since yesterday
Flaky tests	Unstable tests that depend on execution order, timing, or external data
Shared resources	Several tests use the same database and step on each other
Rate limiting	External APIs (GitHub, npm) sometimes block requests that are too frequent

Swipe to see more

How to fix it

Each action below removes a source of randomness identified just before. The first one, pinning dependencies, pays off fastest and comes down to a single command change, shown under the table: going from npm install to npm ci. Never give in on the last row, fixing flaky tests: a flaky test re-run in a loop is not repaired, only hidden, and it will end up masking a real defect.

Action	Why it helps
Pin every dependency	The lock file guarantees the same versions on every run
Isolate tests	Each test creates its own data, no conflict possible
Mock external APIs	Simulate the responses instead of calling the real services
Fix flaky tests	Ignoring them only pushes the problem back

Swipe to see more

# ❌ Non-deterministic
- run: npm install
# Problem: downloads the latest versions, which may have changed

# ✓ Deterministic
- run: npm ci
# npm ci uses exactly the versions from package-lock.json

Common trap

Many teams "solve" flaky tests by re-running them automatically until they pass. The test ends up green, but the defect it reported is still there: you only disabled the alert, you did not fix the cause. The day the same randomness strikes in production, no signal warns you any more.

Ungoverned copy-paste

A pipeline that works gets duplicated from project to project, then diverges with local tweaks. Each copy becomes a one-off, with no common source: a fix no longer spreads, and a copied defect spreads with the template.

Symptom

Each project has its own pipeline, copied from another project, slightly modified. 50 projects = 50 incompatible variants.

project-a/.github/workflows/ci.yml    (original version)
project-b/.github/workflows/ci.yml    (copied, modified)
project-c/.github/workflows/ci.yml    (copied from B, modified again)
project-d/.github/workflows/ci.yml    (copied from C, with C's bugs!)
...

Why it is a problem

The row that really hurts is impossible maintenance. As long as all is well, fifty diverging copies bother nobody. The day a flaw is found in the original template, it has to be fixed by hand in fifty files, hoping not to miss any and not to break anything along the way. Drift is the silent mechanism that makes this fix so risky: each copy has drifted from its starting point, so no fix applies cleanly everywhere.

Problem	Concrete consequence
Inconsistency	One project scans for vulnerabilities, another does not, for no valid reason
Impossible maintenance	A security flaw in the pipeline = 50 files to fix by hand
Drift	Copies diverge over time, nobody knows which version is "the right one"
No capitalisation	One team improves its pipeline, the others do not benefit

Swipe to see more

How to fix it

Centralise the logic in a reusable workflow (workflow_call on GitHub Actions) or a shared template. The principle: a single definition is authoritative, and each project calls it by referencing a version. A fix made to this definition spreads to every project that references it, with no manual copy-paste.

# Before: 200 lines of copy-pasted configuration
# After: 5 lines that call the central template

jobs:
  ci:
    # Referenced by commit SHA, version as a comment, like an action
    uses: org/shared-workflows/.github/workflows/ci.yml@<commit-SHA>  # v1.4.0
    with:
      language: python
      python-version: "3.13"

The shared workflow is referenced like an action: by its commit SHA, with the version as a comment, and not by a tag @v1 that can be moved. The <commit-SHA> placeholder is replaced by the value returned by git ls-remote on the workflows repository.

Result:

• Fixing a bug = changing a single file
• Adding a security check = every project benefits
• New projects start with good practices

Rebuilding for each environment

An artifact is the finished product of your pipeline: a Docker image, an executable, a package. It is what gets deployed.

Symptom

The artifact is rebuilt for each environment. The tell-tale sentence: "It worked in staging!"

staging:
  npm run build        ← Compilation #1
  docker push staging

production:
  npm run build        ← Compilation #2 (potentially different!)
  docker push production

Why it is a problem

It all starts with non-reproducibility. Rebuilding means downloading dependencies again, reusing a different cache, running on a runner in a different state: nothing guarantees that the second compilation produces the same binary as the first. Ghost bugs are the direct consequence, and they are the most expensive to track down, since "it worked in staging" is literally true. You tested one artifact, you deploy another.

Problem	Concrete example
Non-reproducibility	The artifact tested in staging ≠ the artifact deployed in production
Ghost bugs	A bug shows up in production even though "it worked in staging"
Wasted time	Each deployment recompiles everything, even if nothing changed

Swipe to see more

How to fix it

Apply the "Build once, deploy many" principle: a single artifact, promoted from environment to environment.

build:
  npm run build
  docker push app:$COMMIT_SHA    ← Single artifact, identified by its commit

staging:
  docker pull app:$COMMIT_SHA    ← Exactly the same artifact
  ENV=staging deploy         ← Only the configuration changes

production:
  docker pull app:$COMMIT_SHA    ← Still the same artifact
  ENV=production deploy      ← Only the configuration changes

Guarantee: what was tested in staging is exactly what runs in production. The only differences are the environment variables (URLs, API keys, etc.).

The lesson CI/CD best practices details artifact promotion and its prerequisites.

Tests at the end of the chain

Putting the most expensive tests at the very end delays the failure signal until after the build and the deployment. The developer learns about the regression long after moving on, and getting back into context often costs more than the fix itself.

Symptom

Heavy tests (e2e tests, "end-to-end", which exercise the application from end to end through its interface) run after the build and the deployment. Feedback arrives too late.

lint (1s) → build (3min) → deploy (2min) → e2e tests (15min)
                                                    │
                                                    └── Failure detected
                                                        after 20 minutes!

Why it is a problem

The common thread is the cost of waiting. Putting slow tests last is not a problem when they pass; it becomes one when they fail, because all the upstream work (build, deployment) was consumed for nothing. Wasted resources and lost context add up: you burn runner minutes AND you force the developer to dive back into code they left twenty minutes earlier. A test that could have failed in thirty seconds fails after twenty minutes.

Problem	Impact on the developer
Late feedback	20 minutes to find out that a simple bug breaks everything
Wasted resources	Build and deploy run for nothing if the tests fail
Lost context	The developer has started something else and must dive back into the code

Swipe to see more

How to fix it

Apply the test pyramid: run the fast tests first, the slow tests last.

lint (1s) ──┬── unit tests (30s) ──┬── build (3min)
            │                      │
            └── Fast failure!      └── e2e tests (only if everything passes)

The test pyramid visualises this strategy:

         /\           e2e tests (slow, after merge)
        /  \          → Few, check the critical journeys
       /────\
      /      \        Integration tests (on PR)
     /────────\       → Check that components work together
    /          \
   /────────────\     Unit tests (on every commit)
                      → Many, fast, check each function

Logic: the tests at the bottom (unit) are fast and numerous. If one fails, you know within 30 seconds. No need to run the heavy tests.

Dependency on the environment

A runner is the machine (physical or virtual) that runs your pipeline. Different runners can have different configurations.

Symptom

The pipeline works on one specific runner but fails elsewhere. It is the famous "works on my machine", CI edition.

Runner A (ubuntu-22.04): ✓ success
Runner B (ubuntu-24.04): ✗ failure     ← Same code, different result
Runner C (macos): ✗ failure

Common causes

These four causes share a root: the pipeline assumes something about the environment instead of declaring it. Implicit versions are the sneakiest, because they produce no clear error: python simply runs a different version, and your code runs until a subtle incompatibility derails a test. The reflex to adopt when reading this table: each row matches something you thought was a given and that varies from one runner to another.

Cause	Concrete example
Tools not installed	The pipeline uses jq, installed on one runner but not the others
Implicit versions	python calls Python 3.8 on one runner, Python 3.11 on another
Hard-coded paths	/home/runner/specific/path does not exist on every runner
Environment variables	A variable is set on one runner, missing on the others

Swipe to see more

How to fix it

Each fix replaces an assumption with a declaration. The table matches them with the causes above; containerising is the broadest, because the image carries every tool identically everywhere.

Solution	Why it works
Containerise	The Docker image carries every required tool, identical everywhere
Make versions explicit	python-version: "3.13" guarantees the same version on every runner
Avoid absolute paths	$GITHUB_WORKSPACE adapts automatically to the runner
Install prerequisites	If a tool is needed, install it explicitly in the pipeline

Swipe to see more

# ✓ Explicit and portable
permissions: {}                    # ← No rights by default at workflow level
jobs:
  test:
    runs-on: ubuntu-24.04
    permissions:
      contents: read               # ← The bare minimum for this job
    container: python:3.13-slim@sha256:bb2988715db2cf7ace7b53f38f3cffbef7c7046a656bee66245eb0ed386e2e81  # ← Controlled environment
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
        with:
          persist-credentials: false
      - run: pip install -r requirements.txt

Golden rule: if something is not explicitly defined in the pipeline, it can change without warning.

Permanent exceptions

An exemption granted to unblock a release almost never has an end date. Without a deadline or review, it outlives the emergency that justified it, and the scope actually controlled shrinks without anyone having decided so.

Symptom

Projects are exempted from security checks "temporarily". Two years later, the exceptions are still there, and nobody knows whether they are still justified.

Security scan exceptions:
- legacy-project (temporary, added in 2022)    ← Still there
- emergency-project (temporary, added in 2023) ← Still there
- client-X-project (temporary, added in 2023)  ← Still there
- ...

Why it is a problem

The most insidious danger is false security. An all-green dashboard gives the illusion that the chain is healthy, while the exempted projects, invisible in these indicators, are precisely those escaping control. Invisible debt makes it worse: without an inventory of the exceptions and their justification, nobody knows what scope is really protected any more. You think you cover one hundred percent of the estate, you may cover sixty.

Problem	Concrete risk
False security	Dashboards are green, but vulnerable projects slip under the radar
Dangerous precedent	"They have an exception, why not us?", exceptions multiply
Invisible debt	Nobody knows how many projects are exempted nor why

Swipe to see more

How to fix it

The guiding principle: an exception must have a scheduled end, otherwise it is permanent by default. An automatic expiry date is the most effective measure, because it reverses the burden of proof. Instead of requiring someone to remember to remove the exception, its holder must renew it explicitly, with a justification. The other three measures organise traceability around this mechanism and avoid a total bypass when targeted filtering is enough.

Measure	How to apply it
Expiry date	Each exception expires automatically after 90 days
Documented justification	"Why this exception? What is the plan to remove it?"
Regular review	Quarterly audit: exceptions not renewed are removed
Alternative to total bypass	Rather than skipping the whole scan, ignore only the specific vulnerability

Swipe to see more

Good practice

Instead of exempting a project from the security scan, configure the scan to ignore specific vulnerabilities with a justification. For example: "CVE-2023-XXXX ignored because not applicable to our usage (no user input)".

The SPOF pipeline

SPOF means "Single Point of Failure". It is an element whose failure blocks the whole system.

Symptom

A single pipeline does everything, on a single runner, with a single maintainer. If any of these elements goes down, everything stops.

Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

How to fix it

A SPOF is not only technical. The table deliberately separates three kinds of single point, and the most often forgotten one is the single maintainer: a pipeline nobody else understands breaks down as soon as its author goes on leave or changes team. The countermeasure is the same for all three rows, introducing redundancy, whether of machines, trained people or alert channels.

Identified SPOF	Solution
Single runner	Create a pool of runners (at least 2-3)
Single maintainer	Document the pipeline, train at least one other person
No alerts	Set up notifications when the runner is unavailable

Swipe to see more

Audit checklist

Use this list to assess the health of your pipelines. It covers one anti-pattern per line, in the order of the guide. Do not try to tick everything at once: deal first with fast feedback and determinism, which free up the most time for the team. The others get fixed afterwards, once developers have a short and reliable feedback cycle again.

• Fast feedback: duration < 15 min for the first result
• Determinism: no "re-run it, it'll pass"
• Governance: central templates (no copy-paste)
• Build once: artifact built only once
• Test pyramid: fast tests run first
• Traceable exceptions: no exception without an expiry date
• No SPOF: more than one person understands the pipeline

Practical advice

Run this audit once a quarter. Anti-patterns set in gradually, and a regular review catches them before they become critical.

Key points

• An anti-pattern is a solution that creates more problems than it solves
• The monolithic pipeline slows feedback: split it into independent pipelines
• A non-deterministic pipeline erodes trust: pin the dependencies
• Copy-paste multiplies maintenance: centralise in templates
• Build once, deploy many: a single artifact for every environment
• Fast tests first: fail in 30 seconds rather than 20 minutes
• Temporary exceptions become permanent: add expiry dates
• Remove SPOFs: redundancy of runners and of knowledge

Knowledge check

Check that the essentials of this lesson are in place. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions

6 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →