You have CA, certificate, and key files configured for Consul. The agent starts without errors, health checks pass, services register, DNS resolves. But the cluster may still accept plaintext or unauthenticated connections because verify_incoming, verify_outgoing, or verify_server_hostname are missing, set to false, or placed in the wrong configuration stanza.

Certificate files (ca_file, cert_file, key_file) do not enforce TLS on their own. Consul requires explicit verification flags to reject plaintext connections and require client certificates. Without them, any network observer between agents and servers can read the catalog, KV values, and health check results, and any process that can reach the RPC port can impersonate a Consul agent.

This article covers control-plane TLS: the RPC and HTTP API connections between Consul agents and servers. This is distinct from Consul Connect mesh certificates, which govern data-plane mTLS between service sidecars. The two cert lifecycles are independent.

What this means

Three verification flags control whether Consul enforces TLS on its control-plane traffic. Each protects a different trust boundary, and all three must be true for full enforcement.

verify_incoming requires all inbound connections to this agent to use TLS and present a client certificate signed by the configured CA. Without it, the agent accepts plaintext RPC and HTTP connections from any process that can reach the port.

verify_outgoing requires all outbound connections from this agent to use TLS and verify the remote certificate against the CA. Without it, the agent may send traffic in the clear or connect to a server presenting any certificate without verification.

verify_server_hostname verifies that the server certificate matches the expected server.<datacenter>.<domain> pattern. Without it, a stolen certificate from any node can impersonate a server. This flag prevents a compromised client agent from being restarted as a server and receiving replicated cluster state, including ACL tokens and service mesh CA root keys.

flowchart LR
  subgraph Outbound["Agent to server (outgoing)"]
    A1[Agent connects] --> A2{verify_outgoing?}
    A2 -- false --> A3[Plaintext or no cert check]
    A2 -- true --> A4{verify_server_hostname?}
    A4 -- false --> A5[Any node cert passes as server]
    A4 -- true --> A6[Cert must match server.dc.domain]
  end
  subgraph Inbound["Server accepts (incoming)"]
    B1[Connection arrives] --> B2{verify_incoming?}
    B2 -- false --> B3[No client cert required]
    B2 -- true --> B4[Client cert verified against CA]
  end

Since Consul 1.12, these flags must be placed inside the tls stanza rather than at the top level of the configuration file:

tls {
  defaults {
    verify_incoming = true
    verify_outgoing = true
    ca_file   = "/opt/consul/tls/ca.pem"
    cert_file = "/opt/consul/tls/agent.pem"
    key_file  = "/opt/consul/tls/agent-key.pem"
  }

  internal_rpc {
    verify_server_hostname = true
  }
}

verify_server_hostname belongs in tls.internal_rpc, not in tls.defaults. If placed under tls.defaults, it may be silently ignored. Top-level verify_incoming, verify_outgoing, and verify_server_hostname fields are deprecated since 1.12 and emit a warning on agent startup, but operators who carry old configs forward may miss the warning in high-volume logs.

Common causes

CauseWhat it looks likeFirst thing to check
Verify flags absent despite cert filesAgent starts clean, no TLS errors, but consul info shows flags false or missingconsul info | grep -A10 tls on each agent
Top-level verify fields on Consul 1.12+Deprecation warning in startup logs, flags applied to wrong interface or ignoredAgent logs for WARN about deprecated TLS fields
verify_server_hostname in wrong stanzaSet to true under tls.defaults but agent still accepts any node cert as serverConfig file for internal_rpc stanza placement
Client agents with verify_incoming = falseCommon in older setup guides; leaves localhost accepting plaintextClient agent config review
Expired RPC or HTTP certificatesTLS enforced but handshakes fail intermittently or universallyopenssl x509 -enddate -noout -in <cert>

Quick checks

# Check effective TLS configuration on this agent
consul info | grep -A 10 "tls"

# Check certificate expiry for the agent cert
consul tls cert info --cert /opt/consul/tls/agent.pem

# Alternative expiry check if consul tls subcommand is unavailable
openssl x509 -in /opt/consul/tls/agent.pem -noout -enddate

# Validate config file syntax before applying changes
consul validate /etc/consul/

# Check agent startup logs for TLS deprecation warnings
journalctl -u consul --since "24 hours ago" | grep -i "deprecat.*tls"

# Verify the server certificate SAN matches the expected pattern
openssl x509 -in /opt/consul/tls/agent.pem -noout -text | grep -A 1 "Subject Alternative Name"

How to diagnose it

  1. Check effective TLS state on every agent. Run consul info | grep -A 10 "tls" on each server and a sample of client agents. The output shows whether verify flags are true or false at runtime. If any agent shows a flag as false or absent, that agent is not enforcing TLS in that direction.

  2. Check startup logs for deprecation warnings. On Consul 1.12 and later, top-level verify_* fields produce a deprecation warning. If you see these, the flags may not be applied to the correct interface. Grep agent logs for “deprecat” and “tls”.

  3. Verify config stanza placement. Open the config file and confirm that verify_server_hostname = true is inside tls { internal_rpc { ... } }, not inside tls { defaults { ... } }. The verify_incoming and verify_outgoing fields belong in tls { defaults { ... } } unless you need per-interface overrides.

  4. Test enforcement from the network. From a client agent host, attempt a plaintext connection to the server RPC port (default 8300). If the connection succeeds, verify_incoming is not enforced on that server. Use a quick TCP probe, not a sustained connection.

  5. Check certificate expiry. RPC and HTTP control-plane certificates are not exposed as Consul metrics. Run openssl x509 -enddate -noout -in <cert> or consul tls cert info --cert <path> on each agent. Treat any certificate expiring within 7 days as a PAGE-level issue.

  6. Verify the server certificate SAN. The certificate presented by servers must include a SAN entry for server.<datacenter>.<domain>. Without this SAN, verify_server_hostname = true rejects legitimate server connections, which looks like a TLS misconfiguration but is actually a certificate generation problem.

  7. Review client agent configs for verify_incoming = false. Many TLS setup guides leave client agents with verify_incoming = false so that localhost connections (DNS, local API calls) work without TLS. This means the client agent accepts unauthenticated connections on its local interfaces. Evaluate whether this is acceptable for your threat model.

Metrics and signals to monitor

SignalWhy it mattersWarning sign
Effective verify flags (consul info)Confirms TLS enforcement at runtime, not just in configAny flag false or absent on a production agent
Certificate end date (external check)Consul does not expose control-plane cert expiry as a metricExpiry within 7 days is PAGE; within 30 days is PLAN
Agent startup deprecation warningsIndicates top-level TLS fields that may not apply correctlyAny “deprecat” and “tls” log line
RPC error ratePartially enforced TLS can cause intermittent handshake failuresSpike in RPC errors after a config change or cert rotation
Server cert SAN entriesverify_server_hostname requires the correct SAN patternMissing server.dc.domain in SAN list
Gossip encryption key consistencySeparate from TLS but part of the same security postureconsul keyring -list showing mismatched keys

Fixes

Missing or false verify flags

Add verify_incoming = true and verify_outgoing = true inside the tls { defaults { ... } } stanza. After editing, run consul validate on the config directory, then reload the agent.

Warning: enabling verify_incoming on an agent that previously accepted plaintext immediately rejects connections from any client lacking a valid certificate. Stage this change: confirm all connecting clients have valid certs first, then enable the flag in a rolling manner.

Wrong stanza placement on Consul 1.12+

Move verify_server_hostname = true from tls.defaults to tls.internal_rpc. Move any top-level verify_incoming or verify_outgoing into tls.defaults. Run consul validate after editing. Use nested block syntax in HCL files; avoid flattened dotted notation.

Client agents with verify_incoming = false

Evaluate your threat model. If client agents run on hosts where untrusted processes could reach the local API, enabling verify_incoming on clients closes that gap. The tradeoff: all local consumers (DNS, consul-template, health check scripts) must present valid client certificates. This is a coordinated migration, not a single config change.

Expired control-plane certificates

Renew the certificate and key files, then reload or restart the agent. If multiple agents share the same CA but different leaf certs, renew each one. Track renewal dates externally since Consul provides no metric for control-plane cert expiry. Set up automated expiry monitoring using openssl x509 -enddate on a schedule.

Prevention

  • Audit all agents after every config change. Run consul info | grep -A 10 tls on servers and a sample of clients to confirm the effective state matches the intended state.
  • Monitor cert expiry externally. Consul does not expose control-plane cert expiry as a metric. Use file-based or openssl-based checks on a daily schedule. Alert at 30 days, escalate at 7 days.
  • Run consul validate in CI. Catch syntax errors and stanza placement issues before they reach production. Note that consul validate does not catch semantic issues like verify_server_hostname in the wrong stanza.
  • Standardize config templates. Use a single templated config that places all three verify flags in the correct stanzas. Override only where your deployment requires per-interface exceptions.
  • Test from the network. Periodically probe the RPC and HTTP ports with plaintext connections from a host that should not succeed. If it connects, enforcement is broken.
  • Separate control-plane TLS from Connect mesh TLS. Different certificates, potentially different CAs, different expiry timelines. Monitor both independently.

How Netdata helps

  • Certificate file expiry monitoring. Netdata can monitor certificate files on disk and alert on approaching expiry. Since Consul does not expose control-plane cert lifetime as a metric, this fills a gap. Alert at 30 days for planning and 7 days for urgent action.
  • Config file change detection. Netdata tracks file changes, so if an agent config loses a verify flag during a deployment or manual edit, the change is visible alongside Consul’s runtime metrics.
  • Consul RPC and HTTP error correlation. If a partial TLS enforcement change causes intermittent handshake failures, RPC and HTTP error rates spike. Correlating these with config change timestamps narrows the cause.
  • Per-agent process and resource metrics. TLS handshake overhead increases CPU usage on agents. If enabling full verification causes resource pressure, CPU and goroutine metrics on the affected agents show the impact before it cascades.
  • Gossip encryption key consistency. While separate from TLS verification, gossip encryption is part of the same security baseline. Monitoring keyring state across agents catches rotation drift that compounds a TLS enforcement gap.