Operating Vault in production: the complete guide

Running Vault in production goes well beyond installing it. This guide covers the essential operational topics: sizing, high availability, monitoring, backups, maintenance and troubleshooting. Installation, sealing and policies have their own guides, linked below: this one starts where they stop.

This guide does not replace a load test

The sizing and architecture recommendations presented here are orders of magnitude based on field experience. Every environment is different. Validate your configuration with load tests before going to production.

Prerequisites

This guide assumes you have already:

• Installed Vault: see installation and first start, which also covers vault operator init, vault operator unseal and the audit log;
• Understood sealing: the seal and unseal mechanisms are explained in auto-unseal and recovery keys.

What you will learn

• Size a cluster according to your load
• Configure an HA cluster with Raft (Vault's built-in storage, defined just below)
• Set up Prometheus/Grafana monitoring
• Automate Raft backups
• Carry out maintenance operations
• Diagnose common problems
• Apply the production security checklist
• SOCLE anchoring: this guide implements SOCLE-SEC-STO-5 (in French), a hardened vault for sensitive secrets (sealing, audit), with a tested backup on top

Which terms does this page use?

A handful of words come back all along the page, so here they are in one place. Integrated Storage is the storage backend built into Vault, and it relies on the Raft consensus protocol: every node keeps a full copy of the data, and the nodes agree among themselves on the order of writes. The leader (also called the active node) is the only node that accepts writes; the other nodes are standbys (followers), ready to take over if the leader disappears.

The quorum is the majority of the nodes: with 3 nodes, 2 must be reachable for a write to be confirmed. Without a quorum, the cluster stops accepting writes. A performance standby is an Enterprise feature: a standby that can also answer read requests instead of just waiting. A snapshot is a consistent copy of the whole Raft storage at a given moment, taken with the vault operator raft commands (the command family that manages the Raft cluster: peers, snapshots, removal of a node). Telemetry is the set of metrics Vault publishes about its own behaviour (requests, memory, leader changes), which a monitoring tool such as Prometheus collects. A lease is the time-limited right to use a token or a dynamic secret, and its TTL (time to live) is the duration of that right. The seal mechanism is the one that protects the key to Vault's data; see the sealing guide.

Which data sources are critical?

Before going into detail, the official docs identify three pillars for the observability of a Vault cluster. They are complementary: a production cluster must enable all of them. The audit log is set up in the installation guide.

Source	Content	Usage
Audit logs	Every authenticated request	Security, compliance, forensics
Operational logs	System events, errors	Troubleshooting, alerts
Telemetry	Prometheus/StatsD metrics	Monitoring, capacity planning

Swipe to see more

How do I size a Vault cluster?

Two decisions are to be made before any installation, and one is much harder to fix than the other. Adding CPU or RAM to a node is an ordinary operation; changing the number of nodes of a Raft cluster affects the quorum and has to be planned. So settle the topology first, then size the machines.

How many nodes?

The "fault tolerance" column shows how many nodes can disappear without the cluster ceasing to work. Beyond that threshold, Vault does not lose its data but becomes unavailable for writes, for lack of a quorum.

Configuration	Nodes	Fault tolerance	Usage
Development	1	0	Tests, proof of concept
Production	3	1	Starting point for a production
Reinforced tolerance	5	2	Need for extra resilience
Specific cases	7	3	To be justified by the context

Swipe to see more

Why odd numbers?

Raft consensus requires a majority (the quorum). With 3 nodes, 2 must be available. With 5 nodes, 3 must be. An even number does not improve fault tolerance but adds complexity.

Avoid a single node for a genuine production use. A 3-node Raft cluster is often enough: it is the usual starting configuration, and it offers a good balance between resilience and operational simplicity.

What are the orders of magnitude for resources?

The following table gives starting points, not final values. The actual sizing depends on your mix of load. IOPS (input/output operations per second) measure how many reads and writes the disk can absorb each second.

Indicative load	CPU	RAM	Storage	IOPS
Light (< 100 req/s)	2 vCPU	4 GB	25 GB SSD	1000
Medium (100-500 req/s)	4 vCPU	8 GB	50 GB SSD	3000
High (500-2000 req/s)	8 vCPU	16 GB	100 GB SSD	10000
Very high (> 2000 req/s)	16 vCPU	32 GB	200 GB SSD	15000+

Swipe to see more

Validate with a load test

These values are orders of magnitude. HashiCorp recommends tuning and observing your real load rather than relying on fixed tables. Test before deploying.

Which factors drive the sizing?

The number of requests per second is not enough to predict the load, because not every operation costs the same. The factors below explain why two clusters with identical traffic can have very different needs; identify those that match your usage before choosing a row of the previous table. The cryptographic workloads are described in the Transit guide, and the dynamic ones in the dynamic secrets guide.

Factor	Impact
Read/write mix	Writes affect every node (replication)
Transit / PKI	Cryptographic operations = CPU-intensive
Dynamic secrets	More demanding (DB connections, API calls)
Audit logging	Significant I/O depending on volume
Number of secrets	More secrets = more storage and RAM

Swipe to see more

Points of attention:

• Storage: SSD mandatory, NVMe recommended for high loads
• IOPS: often the bottleneck
• Network: latency < 10 ms between cluster nodes
• RAM: Vault keeps a lot in memory for performance

How do I build a highly available cluster?

The built-in Raft storage is today the recommended mode: every node keeps a full copy of the data and the election protocol designates a leader, the only node allowed to write. This architecture removes the dependency on an external storage backend, but it imposes two constraints detailed below: a low-latency network between the nodes and a load balancer able to tell the leader from the followers.

What does a 3-node Raft cluster look like?

The diagram shows the two network flows not to be confused: port 8200 carries the clients' traffic, port 8201 the replication between nodes. They are filtered separately.

Client requests go through port 8200, the Raft links between nodes through port 8201. Source · SVG

Esc to close, + and − to zoom, scroll to move around. 100 % − Fit + Close

How do I configure several Raft nodes?

All the nodes share the same configuration file except for three values, pointed out right after the block. The retry_join parameter deserves particular attention: it lists the other nodes to contact at startup, and Vault keeps retrying as long as none answers, which makes the cluster's startup order irrelevant.

vault.hcl (node 1):

cluster_name = "vault-prod"
cluster_addr = "https://vault-1.internal:8201"
api_addr     = "https://vault-1.internal:8200"

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"

  retry_join {
    leader_api_addr = "https://vault-2.internal:8200"
  }
  retry_join {
    leader_api_addr = "https://vault-3.internal:8200"
  }
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  cluster_addr  = "0.0.0.0:8201"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file  = "/opt/vault/tls/vault.key"
}

seal "awskms" {
  region     = "eu-west-1"
  kms_key_id = "alias/vault-unseal"
}

# Telemetry for Prometheus
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

Adapt for each node:

• cluster_addr and api_addr with the node's address
• a unique node_id
• retry_join towards the other nodes

How does a new node join an existing cluster?

Adding a new node needs no explicit join command if retry_join is set: starting the service is enough. The final check is the only really important step, because a node can start without becoming a voter (a node whose vote counts towards the quorum) in the cluster.

1. Prepare the new node with the same configuration (adapting node_id, cluster_addr, api_addr)

2. Start Vault on the new node:

   sudo systemctl start vault

3. The node contacts the potential leaders through retry_join

4. Check the cluster status:

   vault operator raft list-peers

   Expected output:

   Node       Address                    State       Voter
   ----       -------                    -----       -----
   vault-1    vault-1.internal:8201      leader      true
   vault-2    vault-2.internal:8201      follower    true
   vault-3    vault-3.internal:8201      follower    true

Join behaviour depends on the seal

The join process depends on the seal mechanism:

• Auto-unseal (KMS): the new node joins and unseals itself automatically
• Shamir: the new node receives a challenge and must be unsealed with the unseal keys of the cluster leader to complete the join

With Shamir, the join is not "magic": a manual intervention is needed. The sealing guide explains why every node has to be unsealed individually.

How do I set up the load balancer and its health checks?

The load balancer must spread the traffic and check the health of the nodes. What is specific to Vault is that a healthy node can answer something other than 200: the /v1/sys/health endpoint encodes the state of the node in the HTTP code itself. A balancer left on the default settings will therefore pull every follower node out of the pool, even though they work perfectly.

What are the return codes of /v1/sys/health?

This table is the reference to keep at hand when configuring the probe. Keep in mind that 503 and 501 flag real problems, a sealed node or one that was never initialized, whereas 429 and 473 describe normal operation as a follower.

Node state	Default code	With standbyok=true
Active (leader)	200	200
Standby	429	200
Performance standby (Enterprise)	473	473
Sealed	503	503
Uninitialized	501	501

Swipe to see more

standbyok=true

The standbyok=true parameter makes the standbys answer with the active node's code (200). It is essential for the load balancer to spread traffic over all the nodes, not only the leader. It does not apply to performance standby nodes (Enterprise), which keep code 473: a separate parameter, perfstandbyok=true, exists for them. The table does not list every code: 472 marks a disaster recovery secondary (Enterprise).

HAProxy example:

frontend vault_front
    bind *:443 ssl crt /etc/haproxy/certs/vault.pem
    default_backend vault_back

backend vault_back
    balance roundrobin
    option httpchk GET /v1/sys/health?standbyok=true
    http-check expect status 200

    server vault-1 vault-1.internal:8200 check ssl verify none
    server vault-2 vault-2.internal:8200 check ssl verify none
    server vault-3 vault-3.internal:8200 check ssl verify none

Key points:

• Health check on /v1/sys/health?standbyok=true
• Session persistence not required (stateless)
• The standbys can serve read requests

How do I monitor Vault and raise alerts?

Three events justify waking someone up at night on a Vault cluster: the cluster is sealed, it has no more leader, or the writing of the audit logs fails. In all three cases, the applications that depend on Vault are or will be blocked. The rest of the metrics serves capacity planning and can wait for working hours.

How do I expose the Prometheus metrics?

Vault exposes metrics in Prometheus format on /v1/sys/metrics. Three pieces are needed: enable the telemetry in the Vault configuration, give the scraping token a policy that allows reading the endpoint (the policy language is explained in the policies guide), then declare the Vault nodes as a Prometheus target.

Enable the metrics in vault.hcl:

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
}

Policy to access the metrics:

path "sys/metrics" {
  capabilities = ["read", "list"]
}

read AND list

The list capability is required in addition to read for Prometheus to scrape the metrics properly.

Prometheus scrape config:

scrape_configs:
  - job_name: 'vault'
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    scheme: https
    tls_config:
      insecure_skip_verify: true  # Or configure the CA
    bearer_token_file: /etc/prometheus/vault-token
    static_configs:
      - targets:
        - 'vault-1.internal:8200'
        - 'vault-2.internal:8200'
        - 'vault-3.internal:8200'

Which metrics are worth watching?

This table presents the metrics that are useful first, not absolute thresholds. Adapt the values to your context.

Metric	Suggested alert	Meaning
vault_core_unsealed	== 0 for > 1 min	Vault is sealed
vault_core_active	No node = 1	No leader
vault_raft_leader	Frequent changes	Cluster instability
vault_raft_peers	< expected	Disconnected nodes
vault_audit_log_request_failure	> 0	Audit failing (critical)
vault_runtime_alloc_bytes	Continuous growth	Memory pressure
vault_token_count	To be observed	Potential token leak
vault_expire_num_leases	Fast growth	Leases not revoked

Swipe to see more

Interpret according to usage

A growth of vault_token_count can be normal if your load increases. These metrics must be correlated with your business activity. Consult HashiCorp's complete metrics reference for more details.

Which Prometheus alerts do I recommend?

These five rules cover the events described above, with deliberately different for: durations. A seal is flagged after one minute, whereas the loss of a node waits five minutes: an ordinary restart must not trigger an on-call page. Adapt the memory threshold of the last rule to the RAM actually allocated to your nodes.

groups:
  - name: vault
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault is sealed on {{ $labels.instance }}"

      - alert: VaultNoLeader
        expr: sum(vault_core_active) == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "No active Vault leader"

      - alert: VaultRaftPeersLow
        expr: vault_raft_peers < 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Vault cluster degraded - {{ $value }} nodes"

      - alert: VaultAuditFailure
        expr: rate(vault_audit_log_request_failure[5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Audit log write failure"

      - alert: VaultHighMemory
        expr: vault_runtime_alloc_bytes / (1024*1024*1024) > 12
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Vault is using {{ $value | humanize }}GB of RAM"

Is there a Grafana dashboard for Vault?

The community offers Grafana dashboards for Vault, notably the Vault Metrics Dashboard. A dashboard turns the metrics above into something you can read at a glance during an incident.

Recommended panels:

• Cluster status (leader, peers)
• Requests per second
• Operation latency
• Memory and CPU usage
• Audit log rate
• Token/Lease counts

How do I back up and restore Vault?

A Vault snapshot contains the encrypted data: it is unusable without the sealing mechanism that holds the key. This property is reassuring for confidentiality and formidable for recovery, since backing up Vault without backing up the access to its KMS or its Shamir keys amounts to backing up nothing at all. The following sections cover both aspects, producing the snapshots and then what must go with them for a restore to succeed.

Which types of backups exist?

The edition of Vault determines what you have: the native scheduling of snapshots is an Enterprise feature, while the Community Edition requires you to build the tooling yourself (the editions are compared in the CE vs Enterprise guide). The "export secrets" line is not a backup of the cluster, only a one-off extraction of data.

Type	Content	Availability
Manual Raft snapshot	Complete data	All editions
Automatic Raft snapshot	Complete data, scheduled	Enterprise
Export secrets	Specific secrets	All editions

Swipe to see more

Enterprise automatic snapshots

Vault Enterprise lets you configure automatic snapshots to local or cloud storage (S3, GCS, Azure). In Community Edition, you must automate the manual snapshots with a script and cron.

How do I automate Raft snapshots (CE)?

The automation relies on a dedicated token carrying a policy reduced to reading the snapshot endpoint only. Never use an administration token for this task: it would end up stored in clear text on the machine that runs the scheduled job.

1. Create a backup policy:

   path "sys/storage/raft/snapshot" {
     capabilities = ["read"]
   }

2. Create a dedicated token:

   vault token create \
     -policy=backup \
     -ttl=768h \
     -display-name="backup-automation"

   This service token is not eternal, and renewing it does not change that. Its ceiling is the system maximum TTL (max_lease_ttl, 768 h, i.e. 32 days, by default), counted from its creation and not from the last renewal. A regular vault token renew only pushes the expiry back up to that ceiling: near the limit Vault shortens the granted duration, then the renewal fails. Likewise, -ttl=8760h (one year) would not give you a year of backups.

   The trap is in the error message of the refused renewal:

   Error renewing token: Error making API request.
   URL: PUT http://127.0.0.1:8200/v1/auth/token/renew-self
   Code: 403. Errors:
   * 2 errors occurred:
       * permission denied
       * invalid token

   It mentions neither the TTL nor the ceiling: it is a plain 403, which nothing distinguishes from a permissions problem. The scheduled backup therefore stops silently, and nobody notices until the snapshot is needed. This mechanism was replayed on Vault 2.2.0 with the ceiling reduced to 40 seconds so it could be observed live (creation_time frozen, expire_time pinned to creation plus the ceiling, granted duration shortened, then 403). The behavior over a real 768 h has not been replayed, only the mechanism at reduced scale.

   Two ways to avoid the failure:

   • AppRole, preferred. The backup script authenticates on every run (vault write auth/approle/login) and gets a fresh, short-lived token. No ceiling accumulates, since no token lives long enough to reach it. Two successive logins, spaced beyond the role's ceiling, each returned a fresh token. A role can be created for example like this (adjust the durations):

     vault write auth/approle/role/backup \
       token_policies=backup \
       token_ttl=15m \
       token_max_ttl=1h

     The role's secret_id remains a secret to protect. Its rotation was not measured here; treat it like any static secret.

   • Periodic token, as an alternative. Created with -period=<duration> (creation reserved to root, or to an account whose policy carries create, update and sudo on auth/token/create), it is not subject to the system max_lease_ttl: eight spaced renewals, well beyond the 40 s ceiling of the test, all succeeded. Only an explicit_max_ttl or a revocation stops it. The trade-off: it dies if a single renewal is missed before the end of a period, and, living very long, it is a durable secret to monitor, store carefully and renew.

     Delegating this creation has several traps, measured on Vault 2.2.0. With create and update but no sudo, Vault answers with a 400 (root or sudo privileges required to create periodic token). With sudo alone, without create or update, you get a 403 (permission denied) that seems unrelated: you need the three capabilities together. Finally, a child token can only carry the policies of its parent (otherwise child policies must be subset of parent, also a 400), unless sudo is held. The default policy is attached to the token without being requested, which is surprising when reading the output of vault token lookup.

   Whichever option you pick, a backup that fails must alert, not stay quiet: surface the script failure (exit code, metric, mail) to your monitoring, and also alert when no recent snapshot exists. A backup that was never checked is not a backup.

3. Backup script (/opt/vault/scripts/backup.sh):

   #!/bin/bash
   set -euo pipefail
   
   BACKUP_DIR="/opt/vault/backups"
   RETENTION_DAYS=30
   DATE=$(date +%Y%m%d-%H%M%S)
   BACKUP_FILE="${BACKUP_DIR}/vault-snapshot-${DATE}.snap"
   
   # Check that Vault is reachable
   vault status > /dev/null 2>&1 || {
     echo "ERROR: Vault unreachable"
     exit 1
   }
   
   # Create the snapshot
   vault operator raft snapshot save "${BACKUP_FILE}"
   
   # Check integrity
   if [[ -f "${BACKUP_FILE}" && -s "${BACKUP_FILE}" ]]; then
     echo "SUCCESS: Snapshot created ${BACKUP_FILE}"
   
     # Optional: copy to external storage
     # aws s3 cp "${BACKUP_FILE}" s3://my-vault-backups/
   
     # Clean up old backups
     find "${BACKUP_DIR}" -name "*.snap" -mtime +${RETENTION_DAYS} -delete
   else
     echo "ERROR: Snapshot empty or missing"
     exit 1
   fi

4. Schedule with cron:

   # Every 6 hours
   0 */6 * * * /opt/vault/scripts/backup.sh >> /var/log/vault-backup.log 2>&1

How do I restore from a snapshot?

The restore replaces the entire content of the cluster with that of the snapshot: everything written since the backup was taken is lost, with no confirmation and no way back. It is a crisis manoeuvre, to be rehearsed beforehand on an isolated environment.

Heavy recovery operation

Restoring a Raft snapshot is a recovery operation, not a convenience operation for everyday use. It overwrites all the storage. The exact procedure depends on the recovery scenario (data deleted, logical corruption, loss of the whole cluster).

Follow the official restore documentation for your specific case.

Principle to remember: vault operator raft snapshot restore is a client command. It sends the file to a Vault that is running, unsealed and to which you are authenticated. You must therefore never stop the node before running it. Two situations arise, depending on whether the unseal keys (or the auto-unseal seal) of the current cluster are those of the backup or not.

Case 1: the cluster is running and comes from the same backup (data deleted by mistake, logical corruption). The unseal keys are consistent with the snapshot, so the -force flag is not needed.

1. Check the state: the cluster is initialized, unsealed, and you hold a token allowed to write to sys/storage/raft/snapshot (the read-only backup token is not enough).

2. Copy the snapshot to the machine that runs the client command.

3. Restore on the active node:

   vault operator raft snapshot restore \
     /opt/vault/backups/vault-snapshot-XXXXXX.snap

4. Wait for the restore to really finish: the command returns as soon as Vault has validated the file, before the work is done. Do not resume operations until you have read post-unseal setup complete and vault is unsealed in the log (journalctl -u vault).

5. Check: vault operator raft list-peers, then vault secrets list.

Case 2: the cluster is rebuilt on new storage (all nodes lost). The keys produced by the re-initialization do not match the snapshot data: this is what -force is for.

1. Start one node with empty Raft storage, initialize it with vault operator init and unseal it with the new keys. The root token displayed is temporary: the restore will overwrite it.

2. Authenticate with that temporary root token, and copy the snapshot to the client machine.

3. Force the restore:

   vault operator raft snapshot restore -force \
     /opt/vault/backups/vault-snapshot-XXXXXX.snap

4. Unseal with the original keys: after the restore, the node is sealed. The key holders enter their original keys (those of the backed-up cluster) with vault operator unseal, up to the initial threshold. With auto-unseal, the original KMS or HSM must stay reachable with the same key.

5. Wait for the log (post-unseal setup complete, then vault is unsealed) and check with vault status and vault secrets list.

6. Join the other nodes: with retry_join in their configuration they find the leader on their own; otherwise run vault operator raft join on each one. Check the result with vault operator raft list-peers.

This sequence was replayed on Vault 2.0.3 with a single Raft node: without -force, restoring onto new storage is refused; with -force, it is accepted and the node comes back sealed; it then unseals with the original key, and the backed-up secret is back. A token limited to read on sys/storage/raft/snapshot can take a backup but is refused the restore. The test did not cover a multi-node cluster: joining the other nodes (step 6) follows the HashiCorp documentation.

The -force flag bypasses the check that ensures the unseal keys (Shamir or auto-unseal) are consistent with the snapshot data. It is not a reflex to add everywhere: on a healthy cluster it removes a safeguard. Source: official HashiCorp documentation (the /sys/storage/raft/snapshot-force API and the standard restore procedure). The snapshot does not contain external plugin binaries: copy them into the plugin_directory of every node before restoring.

Why is a snapshot not a complete disaster recovery plan?

A snapshot helps with recovering the data, but a complete Disaster Recovery Plan (DRP) also requires the items below. Enterprise adds DR replication (a secondary cluster kept on standby that continuously receives the primary's data, described in the CE vs Enterprise guide), but even then these items remain yours to provide.

Item	To plan for
Seal mechanism	The KMS/HSM must be reachable
Shamir keys	If used, they must be available
TLS certificates	Backed up separately
Configuration	vault.hcl and scripts
Tested procedure	Monthly restore test

Swipe to see more

Test monthly

A backup that has not been tested is not a backup. Schedule a monthly restore test on an isolated environment.

Which maintenance operations come up regularly?

The four operations below come back regularly in the life of a cluster. Two are done without interrupting the service, the rotation of the internal key and the removal of a node; the other two, regenerating a root token and upgrading, require preparation and a recent snapshot. None should be discovered the day it becomes urgent.

How do I rotate the internal encryption key?

Vault encrypts data with an internal key that can be rotated without downtime (here rotate is the name of the command):

# Check the current key
vault operator key-status

# Perform the rotation
vault operator rotate

# Check the new version
vault operator key-status

What this rotation does (and does not do)

This rotation concerns the internal protection key of Vault's storage. It does not replace:

• The renewal of your business secrets (passwords, API keys...)
• The management of the seal mechanism (KMS, HSM)
• The renewal of dynamic credentials (handled by TTL, the time to live of each lease)

Existing data remains readable. New writes use the new key.

How do I revoke and regenerate the root token?

The initial root token must never be kept. After initialization (see the installation guide):

1. Create admins with appropriate policies

2. Revoke the initial root token:

   vault token revoke s.XXXXX

3. If a new root token is needed (emergency):

   # Start the regeneration (authentication required since Vault 2.0)
   vault operator generate-root -init
   
   # Each key holder provides their share (unseal keys under Shamir,
   # recovery keys under auto-unseal)
   vault operator generate-root \
     -nonce=<nonce> \
     <recovery_key_share>
   
   # Once the threshold is reached, decode the token
   vault operator generate-root \
     -decode=<encoded_token> \
     -otp=<otp>

How do I upgrade Vault?

Vault does not support rolling back to an earlier version: the storage format evolves and an older binary may refuse to read the migrated data. The snapshot taken before the upgrade is therefore your only way back.

Before any upgrade

• Check version compatibility in the release notes
• Test first on a non-production environment
• Plan a maintenance window
• Have a recent snapshot

Package manager

# Update the package
sudo apt update && sudo apt install vault

# Restart (one node at a time in HA)
sudo systemctl restart vault

# Check the version
vault version

Binary

# Download the archive AND the official checksums file
wget https://releases.hashicorp.com/vault/${VERSION}/vault_${VERSION}_linux_amd64.zip
wget https://releases.hashicorp.com/vault/${VERSION}/vault_${VERSION}_SHA256SUMS

# Check the fingerprint of the downloaded archive (must display "OK")
sha256sum --ignore-missing -c vault_${VERSION}_SHA256SUMS

# Stop Vault
sudo systemctl stop vault

# Replace the binary
unzip vault_${VERSION}_linux_amd64.zip
sudo mv vault /usr/local/bin/

# Restart
sudo systemctl start vault

Rolling update procedure (HA cluster):

1. Update a standby (not the leader)

2. Check that it rejoins the cluster and works correctly

3. Repeat for the other standbys

4. Step down the leader (ask it to hand over leadership):

   vault operator step-down

5. Update the former leader, now a standby

6. Check the cluster:

   vault operator raft list-peers
   vault version

How do I remove a node from the cluster?

The removal must be requested from the leader and is done by node_id, not by address. Removing a node lowers the quorum: going from 3 to 2 nodes brings the fault tolerance down to zero, so plan the replacement before the operation.

# Identify the node_id to remove
vault operator raft list-peers

# Remove the node (from the leader)
vault operator raft remove-peer vault-old

How do I troubleshoot a Vault cluster?

Failures met on Vault fall into four families, treated in order of frequency: the service does not start, the cluster is sealed, the nodes no longer agree, or response times degrade. One reflex applies to all of them: vault status first, which tells in a single command whether the node is initialized, sealed, and who the leader is.

What if Vault does not start?

At this stage Vault has not yet opened its API, so the diagnosis goes through the service logs alone. The causes are almost always environment problems, file permissions, a port already taken or an invalid certificate, not problems with Vault's own configuration.

Check the logs:

sudo journalctl -u vault -f

Frequent causes:

Error	Cause	Solution
permission denied	Wrong permissions	chown -R vault:vault /opt/vault
address already in use	Port 8200/8201 taken	ss -tlnp | grep 8200
TLS handshake error	Invalid certificate	Check dates, CA, SAN
KMS access denied	Insufficient IAM	Check the KMS policy
seal configuration missing	Seal config absent	Check vault.hcl

Swipe to see more

Why is Vault unexpectedly sealed?

A sealed node still answers on its API but refuses any operation on the secrets: the decryption key is no longer in memory. What to do depends entirely on the configured mechanism, automatic through a KMS or manual through the Shamir key shares.

# Check the status
vault status

# If auto-unseal, check the KMS access
# AWS
aws kms describe-key --key-id alias/vault-unseal

# If Shamir, unseal manually
vault operator unseal

Causes of sealing:

• Server restart
• Vault crash
• Explicit call to vault operator seal

Loss of the seal mechanism

If Vault restarts while the auto-unseal mechanism is unavailable (KMS/HSM unreachable), it will stay sealed. The lasting loss of the seal mechanism or of its key can make the cluster unrecoverable.

Make sure your seal mechanism is itself highly available. The risks and their mitigations are detailed in the sealing guide.

What if the Raft cluster misbehaves?

These incidents show up as a node missing from the list of peers, or as repeated leader changes. The cause is almost always network and concerns port 8201, distinct from the API port: a firewall that allows 8200 but blocks 8201 leaves nodes reachable by clients while preventing them from forming a cluster.

Check the cluster state:

vault operator raft list-peers

A node does not join:

# Check network connectivity (port 8201)
nc -zv vault-1.internal 8201

# Check the node's logs
sudo journalctl -u vault -f

# Force a new attempt
sudo systemctl restart vault

Split-brain (rare):

If the cluster is partitioned, Raft only elects a leader on the side that keeps the majority of voters. The minority side has no quorum: it can neither elect a leader nor commit writes, and its nodes behave as nodes with no reachable leader. Once connectivity is restored, they find the majority's leader and resynchronize without intervention.

What if performance degrades?

Before concluding that the cluster is undersized, separate reads from writes. A slow read points to the local disk of the node queried; a slow write puts the Raft replication in question, hence the network latency between the nodes, since every write must be confirmed by a majority of them. vault read sys/health is not the tool for that: it describes the state of the node. Latency is read from the telemetry metrics (Raft commit durations, for example) or measured with a network tool between the nodes.

Diagnose:

# Node state (version, sealed, standby, replication): this is not
# a latency measurement
vault read sys/health

# Instant metrics: a token with read and list on sys/metrics
curl -s -H "X-Vault-Token: ${VAULT_METRICS_TOKEN}" \
  "https://vault:8200/v1/sys/metrics?format=prometheus" | grep vault_

Frequent causes:

Symptom	Probable cause	Solution
High read latency	Slow disk	SSD/NVMe, check IOPS
High write latency	Raft replication	Reduce network latency
OOM / crash	Insufficient RAM	Increase RAM, reduce cache
KMS timeouts	Remote/slow KMS	Closer KMS region

Swipe to see more

What if the audit logs stop being written?

This failure takes priority over all the others, because it makes Vault unusable, as the box below explains. The three causes to rule out first are a full disk, incorrect permissions on the destination directory, and a log rotation that moved the expected file (the installation guide shows a rotation configuration that reopens the file correctly).

# Check the audit devices
vault audit list

# Look at the disk space
df -h /var/log

# Check the permissions
ls -la /var/log/vault/

Blocking audit

By default, if all the audit devices fail, Vault refuses all requests. It is a security choice: better a downtime than a hole in the logs.

Which security checklist applies to production?

This checklist serves as a review before going live, then as a periodic check. It is ordered from the most binding to the finest: a failure in the first section makes the cluster vulnerable from day one, whereas the following sections deal with progressive hardening.

Minimal configuration

These five points are the threshold below which a cluster should not receive production traffic. The one most often forgotten is revoking the initial root token, which otherwise remains a total and permanent access, outside any policy control. mlock is the setting that prevents the operating system from writing Vault's memory to disk.

• TLS enabled on all listeners
• Auto-unseal recommended to avoid manual intervention
• Audit device enabled and working
• Root token revoked after initialization
• mlock enabled (except for container constraints)

Shamir can be justified

Auto-unseal is recommended for most production setups, but Shamir can be chosen in some specific contexts (compliance requirements, air-gap...). What matters is understanding the operational implications.

Network

Vault is not meant to be reachable from the Internet: its clients are your applications and your administrators. The two ports are filtered separately, 8200 for clients, 8201 only between the cluster nodes.

• Firewall: ports 8200 (API) and 8201 (cluster) restricted
• Load balancer with /sys/health?standbyok=true health checks
• No direct exposure to the Internet

Policies and authentication

A Vault policy only allows what it explicitly names, everything else is denied. That works in your favour: start from an empty policy and add the paths one by one, rather than removing rights from a policy that is too broad. The policies are covered in the policies guide and the authentication methods in the authentication guide.

• Least-privilege policies for each role
• No root policy for ordinary users
• Suitable auth methods (OIDC, LDAP, Kubernetes)
• Short token TTLs (1h-8h max for interactive use)

Operations

These four points are routine and are checked over time, not at go-live. The restore test is the one that is neglected the fastest, yet it is the only one that proves the backups work.

• Automated backups (daily at minimum)
• Monthly restore test on an isolated environment
• Monitoring with alerts on sealed, no-leader, audit-failure
• Encryption key rotation (vault operator rotate) at least yearly

Secrets

The last level is about how applications use Vault. A perfectly hardened cluster that only hands out static secrets of unlimited lifetime brings hardly more than a protected configuration file. The dynamic secrets guide and the database secrets guide show how to hand out short-lived credentials.

• Dynamic secrets preferred over static ones
• Short TTLs on leases
• Automatic renewal of DB credentials
• Response wrapping for distribution

Key points

1. Cluster of 3 nodes minimum: the base for a resilient production
2. Auto-unseal recommended: avoids manual intervention at restart
3. Monitoring: sealed, no-leader, audit-failure = critical alerts
4. Tested backups: daily Raft snapshot + monthly test
5. Rolling updates: one node at a time, standbys first, check the release notes
6. Audit is mandatory: without audit, you lose traceability
7. Sizing = load test: tables do not replace observing your real load
8. Proof expected by SOCLE: the automatic seal configured, the audit device active and the restore test of a Raft snapshot, all dated, prove the sealing and the audit of SOCLE-SEC-STO-5 (in French); the separation of roles is proven with the policies

Knowledge check

Check that the essentials of this lesson are in place. The questions only cover what has just been explained here.

Knowledge check

Check what you have learned with this interactive quiz

4 questions

4 min

70% to pass

How it works

• The timer starts when you click Start
• Multiple choice, true/false and short answer questions
• You can move between questions
• Detailed results are shown at the end

Start the quiz

Starts the quiz and the timer

×

See in the glossary →