Secret leaks: logs, Docker, variables

"If it is not in Git, it is fine." That is a false idea. Secrets also leak into application logs, Docker images, environment variables, memory dumps and many other places. This guide identifies these leak vectors beyond the source code, sorts them by family and explains the countermeasures that are really reliable depending on the context.

What you will learn

• The three families of leaks: observability, build/artefacts, execution
• How secrets leak into logs, traces and stack traces
• The distinction between build secrets and runtime secrets in Docker
• Why environment variables are more exposed than you think
• The countermeasures for dumps, cloud metadata and local authentication files

Prerequisites

This guide extends the one on secret leaks in code and Git: read it first, because here we deal with the places where a secret leaks outside the repository.

What are the three families of leaks?

Secrets leak in many more places than the source code. To grasp these surfaces better, you can group them into three families:

Observability leaks

Everything that serves to understand the behaviour of the application can capture a secret in passing: logs, distributed traces, stack traces (the list of calls printed when a program fails), support tickets, shared dumps.

Build and artefact leaks

Everything produced while building the software can embed a secret: Docker layers, build caches, CI artefacts, generated configuration files.

Execution leaks

Everything accessible at runtime can expose a secret: environment variables, /proc, cloud metadata (IMDS), process memory.

Which surfaces belong to which family?

This table serves as a risk map: the "Who has access" column is the most instructive, because it shows that a leak almost never stops where it happens. A secret written in a log goes to the aggregator, then to archiving, then to backups. Use it as a review grid, surface by surface, rather than as a simple summary. Two acronyms appear in it: a SIEM (security information and event management) is a platform that collects and analyses security logs, and an APM (application performance monitoring) is a tool that records traces of what applications do.

Family	Surface	How the secret leaks	Who has access
Observability	Application logs	Printed in debug, HTTP traces	Ops, SIEM, archives
Observability	Traces and stack traces	Object serialisation, middleware	Ops, APM, archives
Observability	Tickets and chat	Copy-pasted for debugging	Team, archives
Observability	Shared dumps	Attached to a ticket or sent to a vendor	Support, archives
Build	Docker images	docker history, layers, cache	Registry users
Build	CI artefacts	Build logs, temporary files	CI/CD teams
Execution	Environment variables	/proc/*/environ, dumps, children	Related processes, root
Execution	Memory dumps	Core dumps, heap dumps	Admins, debug
Execution	Cloud metadata	Instance metadata (IMDS)	Any process on the instance
Execution	Local auth files	.npmrc, kubeconfig, Terraform state	Users, CI

Swipe to see more

How do secrets leak through logs and traces?

This family is easy to underestimate, because the tools involved are those meant to help you. A log is designed to be kept, indexed and shared widely: a secret that enters it instantly acquires these three properties. Worse, renewing a secret does not clean the archives: the old value remains readable for as long as the retention allows.

What is the problem?

Secrets leak into logs and traces in many ways, not only through an explicit logger.info:

# ❌ The secret appears in the logs
logger.info(f"Connecting to database with password: {db_password}")

# ❌ The URL contains the password
logger.debug(f"Database URL: {connection_string}")
# Prints: postgresql://user:SuperSecret@db:5432/mydb

But also in less obvious cases:

• HTTP traces: Authorization headers, query strings with tokens
• Debug middleware: raw serialisation of the incoming request
• Stack traces: local variables captured in the exception
• Automatic serialisation: configuration objects logged in full
• Reverse proxy logs: the combined format, the one Nginx applies by default, logs the request line (query string included) along with the Referer and User-Agent headers

Where do logs end up?

A log line does not stay on the server that produced it. It is collected, transported, indexed, then archived, often on several systems whose access rights have nothing to do with those of the original application. The diagram below follows this path.

Application → stdout → Container runtime → Node logs → Aggregator (Loki, ELK)
                                                            ↓
                                                       S3 archiving
                                                            ↓
                                                       Backups

The secret goes through the whole chain and persists in the archives.

How do you stop log leaks?

The defence is built on three levels, from the most reliable to the latest. Writing nothing sensitive remains the only safe measure; application-level masking catches oversights before emission; filtering in the aggregator comes as a last resort, once the line has already left the process. Stack all three, none is enough alone.

1. Never log secrets:

# Good: log without the secret
logger.info("Connecting to database")
logger.info(f"Database host: {db_host}, port: {db_port}")

2. Use masks in logging frameworks:

# Python with structlog
import structlog

def mask_secrets(logger, method_name, event_dict):
    for key in ['password', 'secret', 'token', 'api_key']:
        if key in event_dict:
            event_dict[key] = '***REDACTED***'
    return event_dict

structlog.configure(processors=[mask_secrets, ...])

3. Log aggregator configuration:

# Loki / Promtail: mask the values
scrape_configs:
  - pipeline_stages:
      - replace:
          expression: '(?:password|secret|token)=([^&\s]+)'
          replace: '***REDACTED***'

The replace stage substitutes only the capture group, here the value after the =: the key stays readable (password=***REDACTED***), which helps debugging. The (?:...) group does not capture, so the key is left alone. Promtail reached end of life on 2 March 2026 (Grafana deprecation notice); its successor, Grafana Alloy, offers the same stage.replace stage in its loki.process component.

Observability checklist

To run through during a code review or an acceptance test before going to production. Two of these points, the masks in the aggregator and the reverse proxies, are easy to forget, because they depend not on the application code but on the infrastructure configuration.

• No logger.info/debug with secrets
• Connection URLs without credentials in the logs
• Masks configured in the logging framework
• Masks configured in the log aggregator
• No raw serialisation of configuration objects or HTTP requests
• Reverse proxies configured not to log sensitive headers

How do secrets leak through Docker images and artefacts?

A container image is immutable and distributable: that is its whole point, and it is also what makes a leak impossible to take back. Once the image is pushed to a registry, anyone allowed to download it can extract its layers, including those of an intermediate stage. Fixing the Dockerfile is therefore not enough: you must consider the secret compromised and renew it.

What leaks at build time?

The risk is not limited to docker history. Several mechanisms can capture a secret during the build:

• Command visible in the history: an ARG or ENV with a secret appears in the image metadata
• File copied then deleted: the file stays in an intermediate layer even after RUN rm
• Secret fetched by a tool, then persisted: an npm ci that writes a .npmrc into a final configuration file
• Build cache: layer caches can keep traces

Why are layers the fundamental problem?

Each instruction in a Dockerfile creates a layer. Even if you delete a file, it still exists in the previous layers.

# ❌ The secret is in an intermediate layer
FROM alpine:3.20
COPY .env /app/.env
RUN cat /app/.env && ./setup.sh
RUN rm /app/.env  # The secret is STILL in the previous layer

How is a secret extracted from an image?

These two commands require no particular privilege, only access to the image. The first exposes the Dockerfile instructions as recorded in the metadata, the second unpacks all the layers onto the disk, including those containing files that were later "deleted".

docker history my-image --no-trunc
# Shows all the commands, including secrets in ARG/ENV

docker save my-image | tar -xf -
# Extracts all the layers, including the "deleted" ones

How do build secrets work: --mount=type=secret

Docker provides a dedicated mechanism to expose a secret only during a RUN instruction, without it being recorded in a layer.

# Good: the secret is never in a layer
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN --mount=type=secret,id=db_password \
    cat /run/secrets/db_password | ./setup.sh

docker build --secret id=db_password,src=./db_password.txt .

What do multi-stage builds bring?

A multi-stage build splits the construction into several stages, only one of which produces the final image. The layers of the intermediate stages are not published: only the files explicitly copied with COPY --from= reach the result. Combined with the secret mount, it is the reference setup for installing private dependencies.

# Good: the secret is only in the build stage
FROM node:20@sha256:8f693eaa7e0a8e71560c9a82b55fd54c2ae920a2ba5d2cde28bac7d1c01c9ba5 AS builder
WORKDIR /app
COPY . .
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci

FROM node:20-slim@sha256:2cf067cfed83d5ea958367df9f966191a942351a2df77d6f0193e162b5febfc0
COPY --from=builder /app/dist /app/dist
# The .npmrc is NOT in the final image

Why should you never use ARG for secrets?

ARG is used to parameterise a build, not to inject a confidential value into it: its value is kept in the image metadata and comes out as is with docker history. The frequent workaround of writing the value to a temporary file does not help, since that file in turn creates a layer.

# ❌ ARG is visible in docker history
ARG DB_PASSWORD
RUN echo $DB_PASSWORD > /tmp/setup && ./setup.sh

# Good: use --mount=type=secret
RUN --mount=type=secret,id=db_password \
    cat /run/secrets/db_password | ./setup.sh

What is the difference between build secrets and runtime secrets?

Confusing these two needs leads to useless or dangerous mounts. A build secret only exists for the duration of a RUN instruction and disappears with it; a runtime secret must be available at each container start, throughout its lifetime. No mechanism covers both correctly.

Build secrets ≠ runtime secrets

--mount=type=secret solves build needs: installing private dependencies, accessing a registry, configuring a tool during construction. For the secrets needed at runtime by the application (database password, API key), you need another strategy: volumes, vault, orchestrator or runtime injection.

How do you scan images for secrets?

The last item of the Docker checklist below, the image scan, is much more useful with a concrete example. Trivy has a built-in secrets scanner that analyses the layers of the image:

# Scan an image to detect secrets in the layers
trivy image --scanners secret my-image:1.4.0@sha256:<digest>

# Scan a local directory (before the build)
trivy fs --scanners secret .

The tools themselves are compared in detecting exposed secrets.

Docker checklist

These five points are checked when reviewing a Dockerfile, except the last one, which can be automated in continuous integration. The scan remains essential even if the first four are respected: it catches the secrets that arrived through a dependency or a third-party tool.

• Multi-stage builds everywhere
• --mount=type=secret for build secrets
• Never COPY .env or ARG SECRET
• Build cache cleaned regularly
• Image scan with trivy image --scanners secret

How do secrets leak through environment variables?

The environment variable has become the default injection format, notably under the influence of the twelve-factor methodology (a set of design rules for cloud applications). It solves a portability problem, not a security problem: once set, it is readable by all child processes and by the system's introspection tools. Treating it as a protection is a frequent reasoning error on this subject.

What is the problem: wider visibility and accidental persistence

Environment variables circulate easily between related processes, can reappear in /proc, debug tools, dumps or error handlers, and are therefore often more exposed than you think.

Concretely:

• Child processes inherit the environment at the moment of fork/exec (the system calls that create a new process)
• /proc/<pid>/environ exposes the initial environment of the process (permissions, namespace and ptrace/procfs restrictions determine the real access)
• Debug tools and error handlers can serialise the complete environment
• Crash dumps capture memory, including env vars

# A process with sufficient permissions can read the initial env
cat /proc/1/environ | tr '\0' '\n'
# DB_PASSWORD=SuperSecret

How do they end up in logs and errors?

These two situations are not theoretical: the first comes from a diagnostic script left in place, the second from an error handler configured to attach "the whole context" to the report. In both cases, the value goes to standard output and joins the log collection chain.

# ❌ printenv in a debug script
printenv
# Prints all the secrets

# ❌ Error dump that includes the environment
Exception: Connection failed
Environment: {'DB_PASSWORD': 'SuperSecret', ...}

How do you reduce the exposure of environment variables?

The three measures that follow are not equivalent. Going through a file really changes the exposure model: the content is no longer inherited by child processes and the file system permissions apply. Runtime injection goes further by entrusting the distribution to an agent. The third, reducing the lifetime in memory, is a matter of hygiene and replaces neither of the others.

1. Prefer secret files:

Files rather than env vars: not only Kubernetes

The principle "file rather than env var" applies everywhere, not only in Kubernetes. It is also found in Docker Compose (secrets mounted under /run/secrets/), tmpfs volumes (volumes held in memory, never written to disk), temporary files with restrictive permissions and injection agents such as Vault Agent.

# Kubernetes: volume rather than env
volumes:
  - name: db-creds
    secret:
      secretName: db-credentials
      defaultMode: 0400

containers:
  - name: app
    volumeMounts:
      - name: db-creds
        mountPath: /etc/secrets
        readOnly: true

# Docker Compose: secrets mounted as files
services:
  app:
    # <digest>: your image digest, printed by docker push
    image: my-app:1.4.0@sha256:<digest>
    secrets:
      - db_password
secrets:
  db_password:
    file: ./secrets/db_password.txt
# The secret is mounted in /run/secrets/db_password

# Application: read the file
with open('/etc/secrets/db-password') as f:
    db_password = f.read().strip()

2. Runtime injection with Vault Agent:

template {
  source      = "/etc/vault/templates/db.tpl"
  destination = "/etc/secrets/db.env"
  perms       = 0400
}

3. Reduce the exposure time:

import os

db_password = os.environ.pop('DB_PASSWORD', None)
connect(password=db_password)
db_password = None

Not a guarantee of erasure

Removing the secret from the environment and setting the variable to None reduces the exposure window, but is not a strong guarantee of erasure. Depending on the language, the runtime, the garbage collector (the part of the runtime that frees memory) and the copies in memory, the secret may persist. It is good hygiene, not a sufficient security measure on its own.

Environment variables checklist

The first two points belong to deployment, the next three to code. This separation matters at audit time: fixing the deployment manifest does not protect against a chatty error handler left in the application.

• Critical secrets in files rather than in env vars
• Restrictive permissions on secret files (0400)
• No printenv or env in scripts
• Error handlers that do not print the environment
• No raw serialisation of env vars in error logs

How do secrets leak through dumps and debugging?

A secret always ends up existing in clear text in the process memory, whatever its distribution mode. Any mechanism able to write this memory to disk therefore becomes a leak vector: a core dump after a crash, a heap dump triggered to analyse consumption, a performance profile exposed on an HTTP port. These artefacts are all the more dangerous because they circulate: they are attached to a ticket or sent to a vendor.

Core dumps

A crash can generate a core dump (a file holding a copy of the process memory) containing all of the process memory, including secrets.

Two distinct levels of protection

Disabling core dumps (ulimit -c 0, systemd-coredump configuration) and restricting the container capabilities (securityContext) do not act at the same level. ulimit -c 0 prevents the process from generating the dump. The securityContext reduces the attack surface of the container but does not disable core dumps on its own. Apply both.

Disable the generation of core dumps:

# At the process/shell level
ulimit -c 0

# At the system level (systemd-coredump)
# /etc/systemd/coredump.conf
[Coredump]
Storage=none
ProcessSizeMax=0

Restrict the container capabilities:

# Kubernetes: reduce the attack surface
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

systemd-coredump

On systems using systemd-coredump, a crash summary may be logged and the dump kept depending on the configuration. Check /etc/systemd/coredump.conf to make sure Storage=none is set in production.

Heap dumps (Java, Node.js)

Unlike the core dump, the heap dump is produced on purpose, often in the middle of an incident analysis and under pressure. It contains all the live objects of the process, hence the connection strings, tokens and passwords still referenced.

# ❌ The heap dump contains all the secrets in memory
jmap -dump:format=b,file=heap.hprof <pid>
node --heapsnapshot-signal=SIGUSR2 app.js

The real danger is not only the generation of the dump, but what is done with it afterwards. A dump often becomes a shared artefact: copied off the machine, attached to a ticket, sent to a vendor, stored in a bucket or a shared space, and forgotten there.

Solutions:

• Disable debug endpoints in production
• Restrict access to the dump tools
• Do not persist heap dumps in accessible places
• Treat dumps as sensitive data (encryption, limited retention)

Pprof and profiles

In Go, the anonymous import of net/http/pprof automatically registers its diagnostic routes on the default HTTP server. If that server is the one that exposes your API, the profiles become accessible to anyone who reaches the service, and the /debug/pprof/heap entry point delivers a snapshot of the memory. The remedy is to serve these routes on a separate listener, bound to localhost.

// ❌ pprof endpoint exposed without authentication
import _ "net/http/pprof"

// Good: pprof endpoint on a separate internal port
go func() {
    http.ListenAndServe("localhost:6060", nil)
}()

How do secrets leak through cloud metadata (IMDS)?

The IMDS (Instance Metadata Service) is a local HTTP service, reachable at the address 169.254.169.254, that hands out to applications the credentials of the role attached to the instance. Its particularity: in its original version (IMDSv1), it requires no authentication, the mere ability to send a request from the machine is enough. That is what makes it a frequent target of SSRF flaws (server-side request forgery), where an application is forced to make a request on behalf of the attacker.

What is the risk?

On AWS, GCP and Azure, any process on an instance can potentially retrieve credentials through the metadata service (IMDS):

# AWS IMDSv1: retrieve the IAM role credentials (a single request)
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/my-role

How do you apply defence in depth?

Protection against leaks through IMDS relies on several layers, not on a single mechanism.

1. IMDSv2 mandatory (first measure to apply):

IMDSv2 requires a token obtained through a prior PUT request, which makes exploitation by SSRF much harder.

# Configure the instance to accept only IMDSv2
aws ec2 modify-instance-metadata-options \
  --instance-id i-xxx \
  --http-tokens required

2. Network restrictions as a complement:

NetworkPolicy and IMDS

Blocking 169.254.169.254/32 with a NetworkPolicy is a good complementary practice, but its concrete effectiveness depends on the CNI used (the network plugin of the cluster), the network mode and the routing of traffic to the metadata service. It is not a universal guarantee, it is an extra layer.

# Kubernetes NetworkPolicy (a complement, not the only solution)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-metadata
spec:
  podSelector: {}
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 169.254.169.254/32

3. Partitioning of workloads:

Restrict which pods or processes really need to access the metadata.

4. Prefer workload identities:

Workload identity rather than node identity

The modern goal is often not to "protect IMDS", but to stop depending on it. Workload identities (AWS IRSA, GCP Workload Identity, Azure Workload Identity) attach credentials directly to the pod/service, without going through the node's IAM role. It is safer and more granular.

The implementation details are in the guide Machine identity and workload identity.

Which local authentication files are at risk?

A surface often forgotten: the authentication files present on development workstations and CI runners. These files contain tokens, credentials or sensitive state:

File	Sensitive content
.npmrc	Private npm registry token
.pypirc	PyPI credentials
~/.docker/config.json	Docker registry tokens
~/.kube/config	Kubernetes certificates and tokens
terraform.tfstate	Terraform state (may contain passwords, keys)
~/.aws/credentials	AWS access keys
~/.config/gcloud/	Google Cloud credentials

Swipe to see more

Good practices for auth files

• Never commit these files (check your .gitignore)
• Prefer temporary authentication mechanisms (OIDC, SSO)
• On CI runners, use ephemeral secrets injected by the orchestrator
• Check permissions: 0600 at most on these files

What should you not conclude from this guide?

This guide shows that secrets leak well beyond the source code. But be careful not to draw false conclusions from it:

• Not putting a secret in Git is not enough, it can leak through logs, images or the environment
• Putting a secret in an environment variable does not secure it, it is a distribution mode, not a protection measure
• Putting a secret in a Kubernetes Secret does not make it a vault, it is a base64-encoded object, not encrypted by default
• Deleting a file from a final image does not erase its trace from the build, the intermediate layers persist
• Disabling a debug endpoint does not protect the dump already generated, the artefacts may have been copied elsewhere

Summary table

Here are the nine surfaces treated in detail in this guide, with their main countermeasure, to use as a review aid rather than as linear reading. The opening table listed ten: tickets and shared dumps appear there as observability surfaces, but they come down to one instruction, never paste a secret into them, and the circulation of dumps is covered by the heap dumps row below. One remark while going through it: no row is settled by a single tool, each combines a design decision and an operations setting.

Family	Surface	Risk	Countermeasure
Observability	Logs	Long-term archiving	Masks, do not log secrets
Observability	Traces and stack traces	Automatic serialisation	Header filtering, masking
Build	Docker layers	docker history, extraction	Multi-stage, --mount=type=secret
Build	Cache and artefacts	Persistence of the secret	Cache cleaning, Trivy scan
Execution	Env variables	/proc, inheritance, dumps	Files, runtime injection
Execution	Core dumps	Complete memory	ulimit -c 0, Storage=none
Execution	Heap dumps	Shared artefact	Disable in prod, retention
Execution	IMDS	Node credentials	IMDSv2, workload identity
Execution	Auth files	Permanent tokens	Permissions, ephemeral auth

Swipe to see more

Key points

1. Leaks outside Git fall into three families, observability, build/artefacts, execution, each with its own countermeasures
2. Distinguish build secrets from runtime secrets, --mount=type=secret protects the build, not the execution
3. Docker layers keep everything, multi-stage and secret mounts are essential
4. Environment variables are more exposed than you think, prefer files with restrictive permissions
5. Dumps become shared artefacts, treat them as sensitive data
6. IMDS is protected in depth, IMDSv2, network restrictions, partitioning, and above all workload identities
7. Local auth files are a forgotten surface, .npmrc, kubeconfig, Terraform state can all leak

Knowledge check

Check that the essentials of this lesson are in place. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

4 questions

4 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →