Vault

Kind: vault

Overview

Netdata can pull collector credentials directly from HashiCorp Vault at runtime, so you never store passwords or tokens in plain-text configuration files.

This page covers Vault specific setup. For the full resolver overview and syntax reference, including simpler alternatives like ${env:...}, ${file:...}, and ${cmd:...}, see Secrets Management.

Limitations

Netdata reads existing secrets from Vault. It does not create or renew Vault tokens. If the configured token expires or becomes invalid, secret resolution fails until Netdata can read a valid token again. If you use token_file mode, Netdata re-reads the file on every secret resolution, so an external process (e.g. Vault Agent, a cron job) can renew the token by writing to the file. For KV v2 secrets, Netdata does not add /data/ to the path automatically.

The Dynamic Configuration Test action initiates one real authenticated GET /v1/auth/token/lookup-self request using the configured token, namespace header, TLS settings, proxy path, and timeout. HTTP 200 is operational success by status; its response body is not inspected. An exact permission-only HTTP 403 body is reported as validation-only because it cannot reliably distinguish a valid token without self-lookup permission from an invalid token on older Vault versions or another authentication restriction. Invalid-token, ambiguous, malformed, or oversized HTTP 403 bodies fail, as does every other HTTP status.

The request does not read a secret or prove access to any secret path. It also cannot prove that an intermediary preserved the namespace header or that Vault used it. A gateway or allowlist can block the self-lookup route even when configured secret reads would work. When the request reaches Vault, it creates normal audit and activity evidence. For a limited-use service token, Test can consume the final remaining use and cause later secret resolution to fail; an external proxy or service mesh can also retry independently.

A configured token file must resolve to a regular file no larger than 1 MiB. Symlinks to regular files are supported.

Setup

You can configure the vault secretstore in two ways:

Prerequisites

Make Vault reachable

Netdata must be able to reach your Vault server at the address you configure in addr.

Provide a Vault token

Choose one supported authentication mode and make sure Netdata can use it:

• token: store the Vault token directly in the secretstore configuration.
• token_file: store the Vault token in a local file on the Netdata host that is readable by the netdata user.

Prefer token_file for production so the Vault token is not embedded directly in the secretstore configuration.

Allow access to the referenced secret paths

The Vault token used by this secretstore must have a policy that grants read capability on the paths you reference from collector configs. Scope the policy to only the paths Netdata needs.

Plan for file-based changes

If you edit /etc/netdata/go.d/ss/vault.conf, restart the Netdata Agent to load the updated secretstore definition.

Configuration

Options

The following options can be defined for this secretstore backend.

mode

Supported values:

• token: store the Vault token directly in the secretstore configuration.
• token_file: read the Vault token from a local file on the Netdata host.

Prefer token_file for production so the token is stored separately from the secretstore configuration.

tls_skip_verify

This is insecure. Use it only as a temporary workaround or in a non-production environment.

mode_token_file.path

The path may be a regular file or a symlink to a regular file. The file must be no larger than 1 MiB.

via UI

1. Open the Netdata Dynamic Configuration UI.
2. Go to Collectors -> go.d -> SecretStores -> vault.
3. Add a new secretstore and give it a store name.
4. Fill in the backend-specific settings.
5. Save the secretstore.

via File

Define the secretstore in /etc/netdata/go.d/ss/vault.conf.

Each file contains a jobs array, and the secretstore kind is determined by the filename.

After editing the file, restart the Netdata Agent to load the updated secretstore definition.

Examples

Token

Store the Vault token directly in the secretstore definition.

jobs:
  - name: vault_prod
    mode: token
    mode_token:
      token: your-vault-token
    addr: https://vault.example

Token from environment variable

Use a ${env:...} resolver for the Vault token to avoid storing it in plain text in the secretstore config file.

jobs:
  - name: vault_prod
    mode: token
    mode_token:
      token: "${env:VAULT_TOKEN}"
    addr: https://vault.example

Token file

Read the Vault token from a local file on the Netdata host. Netdata re-reads the file on every secret resolution, so an external process can renew the token by writing to this file.

jobs:
  - name: vault_prod_file_token
    mode: token_file
    mode_token_file:
      path: /var/lib/netdata/vault.token
    addr: https://vault.example

Vault Enterprise with namespace

Connect to a Vault Enterprise server using a specific namespace.

jobs:
  - name: vault_enterprise
    mode: token_file
    mode_token_file:
      path: /var/lib/netdata/vault.token
    addr: https://vault.example
    namespace: admin

Use in collector configs

Use the ${store:vault:...} syntax to reference Vault secrets in any string field of a collector configuration file.

The operand is path#key.

Netdata sends the path to Vault as /v1/<path> exactly as you provide it. For KV v2 secrets, include /data/ in the path yourself. The path must not be empty and must not contain .., ?, or #.

• KV v1 example: ${store:vault:vault_prod:secret/netdata/mysql#password}.
• KV v2 example: ${store:vault:vault_prod:secret/data/netdata/mysql#password} — note the /data/ segment, Netdata does not add it automatically.

${store:vault:<store-name>:<path#key>}

• vault: The secretstore backend kind.
• <store-name>: The name of the configured secretstore, for example vault_prod.
• <path#key>: The Vault API path and the secret field name, separated by #.

Examples

MySQL collector with password from Vault

This example configures a MySQL collector job in /etc/netdata/go.d/mysql.conf. The password in the DSN connection string is not stored in plain text. Instead, ${store:vault:vault_prod:secret/data/netdata/mysql#password} tells Netdata to read the KV v2 secret at secret/data/netdata/mysql from the vault_prod store, extract the password field from the response, and substitute it into the DSN at runtime.

# /etc/netdata/go.d/mysql.conf
jobs:
  - name: mysql_prod
    dsn: "netdata:${store:vault:vault_prod:secret/data/netdata/mysql#password}@tcp(127.0.0.1:3306)/"

Elasticsearch collector with HTTP basic auth from Vault

This example configures an Elasticsearch collector job in /etc/netdata/go.d/elasticsearch.conf. The password field uses a secret reference instead of a plain-text value. Netdata reads the KV v2 secret at secret/data/netdata/elasticsearch from the vault_prod store, extracts the password field, and substitutes it at runtime.

# /etc/netdata/go.d/elasticsearch.conf
jobs:
  - name: es_prod
    url: https://elasticsearch.example.com:9200
    username: netdata
    password: "${store:vault:vault_prod:secret/data/netdata/elasticsearch#password}"

Troubleshooting

Find the exact error

Check the Netdata Agent logs when the collector starts or restarts. Vault resolver errors include messages such as vault returned HTTP 403, vault path contains invalid characters, operand must be in format 'path#key', or key 'password' not found in vault response.

Vault returns permission denied or the token has expired

Check the Vault token policy and, if you use Vault Enterprise namespaces, confirm that namespace is correct. If you use a short-lived token, make sure the token is renewed or replaced before it expires.

The Dynamic Configuration Test fails

The Test action makes a real authenticated request to Vault’s token self-lookup endpoint. HTTP 200 is operational success by status; its response body is not inspected. An exact permission-only HTTP 403 body is reported as validation-only because it cannot reliably prove whether the token is valid. Invalid-token, ambiguous, malformed, or oversized HTTP 403 bodies fail. An unreachable endpoint and every other HTTP status also fail.

Operational success proves only that the responding endpoint returned HTTP 200 for self-lookup at that time. It does not prove that Vault accepted the token or that the token can read a particular secret path. Check secret-path policies separately. A gateway or deployment allowlist can reject the self-lookup route even when configured secret reads would work.

Secret or key is not found

Check the operand carefully:

• Make sure the path is the Vault API path.
• For KV v2, make sure the path includes /data/.
• Make sure the key exists in the returned secret payload.

TLS verification fails

Make sure the Netdata host trusts the CA that signed the Vault certificate. Use tls_skip_verify: true only as an insecure workaround.

Token file cannot be read

Check the file path, file contents, and that the netdata user can read the file.