$ guides / consul / consul-tls-verify-not-enforced ▌

Operations Guides

Consul TLS not fully enforced: verify_incoming, verify_outgoing, and server hostname

You have CA, certificate, and key files configured for Consul. The agent starts without errors, health checks pass, services register, DNS resolves. But the cluster may still accept plaintext or unauthenticated connections because verify_incoming, verify_outgoing, or verify_server_hostname are missing, set to false, or placed in the wrong configuration stanza.

Certificate files (ca_file, cert_file, key_file) do not enforce TLS on their own. Consul requires explicit verification flags to reject plaintext connections and require client certificates. Without them, any network observer between agents and servers can read the catalog, KV values, and health check results, and any process that can reach the RPC port can impersonate a Consul agent.

This article covers control-plane TLS: the RPC and HTTP API connections between Consul agents and servers. This is distinct from Consul Connect mesh certificates, which govern data-plane mTLS between service sidecars. The two cert lifecycles are independent.

What this means

Three verification flags control whether Consul enforces TLS on its control-plane traffic. Each protects a different trust boundary, and all three must be true for full enforcement.

verify_incoming requires all inbound connections to this agent to use TLS and present a client certificate signed by the configured CA. Without it, the agent accepts plaintext RPC and HTTP connections from any process that can reach the port.

verify_outgoing requires all outbound connections from this agent to use TLS and verify the remote certificate against the CA. Without it, the agent may send traffic in the clear or connect to a server presenting any certificate without verification.

verify_server_hostname verifies that the server certificate matches the expected server.<datacenter>.<domain> pattern. Without it, a stolen certificate from any node can impersonate a server. This flag prevents a compromised client agent from being restarted as a server and receiving replicated cluster state, including ACL tokens and service mesh CA root keys.

flowchart LR
  subgraph Outbound["Agent to server (outgoing)"]
    A1[Agent connects] --> A2{verify_outgoing?}
    A2 -- false --> A3[Plaintext or no cert check]
    A2 -- true --> A4{verify_server_hostname?}
    A4 -- false --> A5[Any node cert passes as server]
    A4 -- true --> A6[Cert must match server.dc.domain]
  end
  subgraph Inbound["Server accepts (incoming)"]
    B1[Connection arrives] --> B2{verify_incoming?}
    B2 -- false --> B3[No client cert required]
    B2 -- true --> B4[Client cert verified against CA]
  end

Since Consul 1.12, these flags must be placed inside the tls stanza rather than at the top level of the configuration file:

tls {
  defaults {
    verify_incoming = true
    verify_outgoing = true
    ca_file   = "/opt/consul/tls/ca.pem"
    cert_file = "/opt/consul/tls/agent.pem"
    key_file  = "/opt/consul/tls/agent-key.pem"
  }

  internal_rpc {
    verify_server_hostname = true
  }
}

verify_server_hostname is inherited from tls.defaults unless overridden, but HashiCorp recommends placing it explicitly in tls.internal_rpc; valid nested locations are tls.defaults and tls.internal_rpc. Top-level verify_incoming, verify_outgoing, and verify_server_hostname fields are deprecated since 1.12 and emit a warning on agent startup, but operators who carry old configs forward may miss the warning in high-volume logs.

Common causes

Cause	What it looks like	First thing to check
Verify flags absent despite cert files	Agent starts clean, no TLS errors, but `consul info` shows flags false or missing	`consul info \| grep -A10 tls` on each agent
Top-level verify fields on Consul 1.12+	Deprecation warning in startup logs, flags applied to wrong interface or ignored	Agent logs for WARN about deprecated TLS fields
`verify_server_hostname` in wrong stanza	Set to true under `tls.defaults` but agent still accepts any node cert as server	Config file for `internal_rpc` stanza placement
Client agents with `verify_incoming = false`	Common in older setup guides; leaves localhost accepting plaintext	Client agent config review
Expired RPC or HTTP certificates	TLS enforced but handshakes fail intermittently or universally	`openssl x509 -enddate -noout -in <cert>`

Quick checks

# Inspect the running agent's full runtime TLS configuration
curl -s http://127.0.0.1:8500/v1/agent/self | jq '.DebugConfig.TLS // .DebugConfig'

# Check certificate expiry (the TLS CLI creates certificates; it does not inspect arbitrary certs)
openssl x509 -in /opt/consul/tls/agent.pem -noout -enddate

# Validate config file syntax before applying changes
consul validate /etc/consul/

# Check agent startup logs for TLS deprecation warnings
journalctl -u consul --since "24 hours ago" | grep -i "deprecat.*tls"

# Verify the server certificate SAN matches the expected pattern
openssl x509 -in /opt/consul/tls/agent.pem -noout -text | grep -A 1 "Subject Alternative Name"

How to diagnose it

Check effective TLS state on every agent. Inspect /v1/agent/self DebugConfig (or the effective TLS lines emitted when the agent starts) on each server and a sample of clients. consul info summarizes Stats only and does not reliably print these verify flags. If the effective value is false or absent for an interface, that interface is not enforcing TLS in that direction.
Check startup logs for deprecation warnings. On Consul 1.12 and later, top-level verify_* fields produce a deprecation warning. If you see these, the flags may not be applied to the correct interface. Grep agent logs for “deprecat” and “tls”.
Verify config stanza placement. Open the config file and confirm that verify_server_hostname = true is either inherited from tls { defaults { ... } } or, preferably, set explicitly in tls { internal_rpc { ... } }; it is invalid in per-interface tls.https or tls.grpc stanzas. The verify_incoming and verify_outgoing fields belong in tls { defaults { ... } } unless you need per-interface overrides.
Test enforcement from the network. A TCP connect to port 8300 can succeed even when TLS is enforced, because the listener accepts the socket first. From a host without a client certificate, run a TLS probe such as openssl s_client -connect <server>:8300 and confirm that the server rejects the handshake; a completed handshake without a client certificate means verify_incoming is not enforced.
Check certificate expiry. RPC and HTTP control-plane certificates are not exposed as Consul metrics. Run openssl x509 -enddate -noout -in <cert> or consul tls cert info --cert <path> on each agent. Treat any certificate expiring within 7 days as a PAGE-level issue.
Verify the server certificate SAN. The certificate presented by servers must include a SAN entry for server.<datacenter>.<domain>. Without this SAN, verify_server_hostname = true rejects legitimate server connections, which looks like a TLS misconfiguration but is actually a certificate generation problem.
Review client agent configs for verify_incoming = false. Many TLS setup guides leave client agents with verify_incoming = false so that localhost connections (DNS, local API calls) work without TLS. This means the client agent accepts unauthenticated connections on its local interfaces. Evaluate whether this is acceptable for your threat model.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
Effective verify flags in `/v1/agent/self` `DebugConfig` or startup output	Confirms TLS enforcement at runtime, not just in config	Any effective flag false or absent on a production agent
Certificate end date (external check)	Consul does not expose control-plane cert expiry as a metric	Expiry within 7 days is PAGE; within 30 days is PLAN
Agent startup deprecation warnings	Indicates top-level TLS fields that may not apply correctly	Any “deprecat” and “tls” log line
RPC error rate	Partially enforced TLS can cause intermittent handshake failures	Spike in RPC errors after a config change or cert rotation
Server cert SAN entries	`verify_server_hostname` requires the correct SAN pattern	Missing `server.dc.domain` in SAN list
Gossip encryption key consistency	Separate from TLS but part of the same security posture	`consul keyring -list` showing mismatched keys

Fixes

Missing or false verify flags

Add verify_incoming = true and verify_outgoing = true inside the tls { defaults { ... } } stanza. After editing, run consul validate on the config directory, then reload the agent.

Warning: enabling verify_incoming on an agent that previously accepted plaintext immediately rejects connections from any client lacking a valid certificate. Stage this change: confirm all connecting clients have valid certs first, then enable the flag in a rolling manner.

Wrong stanza placement on Consul 1.12+

Move verify_server_hostname = true explicitly to tls.internal_rpc (it may also be inherited from tls.defaults, but explicit placement avoids ambiguity). Move any top-level verify_incoming or verify_outgoing into tls.defaults. Run consul validate after editing; its builder emits messages such as The 'verify_server_hostname' field is deprecated. Use the 'tls.internal_rpc.verify_server_hostname' field instead. Use nested block syntax in HCL files and treat validation plus startup logs as necessary but not sufficient evidence of enforcement.

Client agents with verify_incoming = false

Evaluate your threat model. If client agents run on hosts where untrusted processes could reach the local API, enabling verify_incoming on clients closes that gap. The tradeoff: all local consumers (DNS, consul-template, health check scripts) must present valid client certificates. This is a coordinated migration, not a single config change.

Expired control-plane certificates

Renew the certificate and key files, then reload or restart the agent. If multiple agents share the same CA but different leaf certs, renew each one. Track renewal dates externally since Consul provides no metric for control-plane cert expiry. Set up automated expiry monitoring using openssl x509 -enddate on a schedule.

Prevention

Audit all agents after every config change. Inspect /v1/agent/self DebugConfig (or restart-time TLS output) on servers and a sample of clients to confirm the effective state matches the intended state.
Monitor cert expiry externally. Consul does not expose control-plane cert expiry as a metric. Use file-based or openssl-based checks on a daily schedule. Alert at 30 days, escalate at 7 days.
Run consul validate in CI. Catch syntax errors and stanza placement issues before they reach production. Note that consul validate does not catch semantic issues like verify_server_hostname in the wrong stanza.
Standardize config templates. Use a single templated config that places all three verify flags in the correct stanzas. Override only where your deployment requires per-interface exceptions.
Test from the network. Periodically probe the RPC and HTTP ports with plaintext connections from a host that should not succeed. If it connects, enforcement is broken.
Separate control-plane TLS from Connect mesh TLS. Different certificates, potentially different CAs, different expiry timelines. Monitor both independently.

How Netdata helps

Certificate file expiry monitoring. Netdata can monitor certificate files on disk and alert on approaching expiry. Since Consul does not expose control-plane cert lifetime as a metric, this fills a gap. Alert at 30 days for planning and 7 days for urgent action.
Config file change detection. Netdata tracks file changes, so if an agent config loses a verify flag during a deployment or manual edit, the change is visible alongside Consul’s runtime metrics.
Consul RPC and HTTP error correlation. If a partial TLS enforcement change causes intermittent handshake failures, RPC and HTTP error rates spike. Correlating these with config change timestamps narrows the cause.
Per-agent process and resource metrics. TLS handshake overhead increases CPU usage on agents. If enabling full verification causes resource pressure, CPU and goroutine metrics on the affected agents show the impact before it cascades.
Gossip encryption key consistency. While separate from TLS verification, gossip encryption is part of the same security baseline. Monitoring keyring state across agents catches rotation drift that compounds a TLS enforcement gap.