Apache configtest and Include wildcards: catching bad config before it bites

apachectl configtest (equivalent to httpd -t) parses your Apache configuration and reports Syntax OK or a specific error. It is the cheapest safety check in your toolchain, and it is the main thing standing between a bad edit and the stale-config trap: the state where you believe a new configuration is live, but Apache rejected it at reload time and is still running the old one.

The trap works like this. You edit a config file, run a graceful reload, and move on. If the reload fails the config check, the old configuration keeps running and the new one is discarded. Nothing in the process table or access log tells you. The only evidence is a few lines in the error log that nobody greps. Hours later, someone restarts Apache for an unrelated reason and the server fails to come up, because the config on disk was never valid.

This guide covers what configtest actually validates, the Include wildcard behaviors that quietly change which files Apache reads, the warnings that mask real problems, and how to wire configtest into your reload path so bad config never reaches a running server.

What configtest validates, and what it does not

apachectl configtest and httpd -t parse the full configuration tree and report syntax errors, unresolved directives, and structural problems. Exit code 0 on success, non-zero on failure. On Windows the equivalent is httpd.exe -t.

What it catches:

• Syntax errors in the main config and in any file pulled in by Include or IncludeOptional.
• Directives from modules that are not loaded, which show up as unknown directive errors.
• SSL certificate and key path or format problems referenced in the config.
• Structural mistakes: unclosed containers, directives in the wrong context, malformed VirtualHost blocks.

What it does not catch:

• Runtime behavior. A config can pass configtest and still fail under traffic: backend unreachable, permissions wrong on a document root, a handler that crashes on the first request.
• Whether the server is running or serving anything. apachectl -t validates syntax only. It proves nothing about the live process.
• Problems that only appear when new children spawn. A graceful reload can succeed at the parent level while new-generation children fail to start on runtime errors the syntax check never saw, leaving the old children to carry the load on a shrinking pool.

Treat configtest as a gate, not a guarantee. It answers one question: will Apache accept this configuration tree? Ask it before every reload. The cost of asking is a fraction of a second; the cost of a wrong answer is the next restart.

The stale-config trap

The reload path is where config errors do the most damage, and the behavior is not uniform.

Upstream, apachectl restart and apachectl graceful run configtest before acting, so a broken config stops the restart rather than killing the running server. But whether a given distribution’s graceful path checks config before reloading varies, and plenty of reloads never go through apachectl at all: logrotate scripts, configuration management, systemd reload units, and kill -USR1 all bypass whatever wrapper logic exists.

When a direct kill -USR1 reload fails the config check, the running configuration stays, the new one is discarded, and the record is in the error log; a wrapper that runs the test first can stop before signaling. Detect it directly:

# Confirm current config is valid right now
apachectl configtest 2>&1

# Look for failed reloads versus successful ones
grep -iE "resuming normal operations|syntax error|configuration error" \
  /var/log/apache2/error.log 2>/dev/null | tail -10
# RHEL path: /var/log/httpd/error_log

If you see a syntax error timestamped after your last change and no “resuming normal operations” after it, the running server is on stale config. Fix the file, re-run configtest, and reload again. Do not assume the next deploy will sort it out: the next deploy edits the same broken file and fails the same way.

A second, quieter variant: the syntax check passes but runtime errors prevent new children from starting after a graceful restart. Old children keep serving on the old config while the pool drains. The tell is repeated child startup failures in the error log around a reload, with no matching outage in the access log. This is why “configtest passed” is necessary but never sufficient evidence that a reload landed.

Include wildcards: the quiet footgun

Include and IncludeOptional accept wildcards, which is how most distributions wire up sites-enabled/*.conf and conf.d/*.conf. Two behaviors matter operationally.

Wildcard directories accept anything. Any file matching the pattern is parsed, in full, as Apache configuration. A .conf file dropped into an included directory by a package install, a config management run, a careless cp, or a leftover from a decommissioned vhost becomes live configuration on the next reload. Nothing warns you that a file you did not write is now part of the server. The set of files Apache reads is whatever happens to be on disk, not whatever you think you deployed. Periodically audit what the wildcards actually resolve to:

# See every file a given Include wildcard will pull in
ls -1 /etc/apache2/sites-enabled/*.conf 2>/dev/null
ls -1 /etc/httpd/conf.d/*.conf 2>/dev/null

Zero matches behave differently for the two directives. Include with a wildcard that matches nothing is a hard failure: configtest errors out and the server will not start or reload. IncludeOptional with a non-matching wildcard is silently ignored. The same applies to non-existent paths: IncludeOptional ignores them (on 2.4.30 and later for plain paths without wildcards; older releases could still error). The practical rule: use Include only when a missing file should be fatal because the config is meaningless without it, and IncludeOptional for optional drop-in directories that may legitimately be empty. An empty sites-enabled directory with a plain Include will take your server down on the next restart.

To see the configuration Apache actually resolved, with all includes expanded, dump the pre-parsed tree:

# Dump the fully resolved configuration (mod_info required)
httpd -DDUMP_CONFIG -k start 2>/dev/null | head -50

This is the fastest way to answer “which of these files is actually in effect” when a vhost behaves unexpectedly. It shows the config as Apache sees it, not as you organized it.

Warnings that mask real problems

Configtest output mixes fatal errors with warnings, and the warnings train people to stop reading. Two regulars:

• AH00548: NameVirtualHost has no effect and will be removed in the next release. The directive is deprecated; since 2.3.11 Apache enables name-based virtual hosting automatically when multiple vhosts share an address/port, and the directive currently has no effect. Remove the line: it is noise that hides real output.
• AH01574: module security2_module is already loaded, skipping. Usually a duplicate LoadModule line, often from a wildcard include pulling the same module config twice. Harmless in itself, but it tells you your include tree has redundancy, and it sits in the same output stream as genuine syntax errors.

The failure mode is not the warnings; it is operator habit. When every configtest prints three lines of ignorable warnings, the fourth line, the real error, gets skimmed past. Hold a standard of zero errors and zero warnings from configtest, so that any output at all is signal.

Making configtest a pre-reload gate

The fix for the stale-config trap is procedural: never reload without a passing configtest immediately before it, in the same shell or the same automation step.

flowchart TD
    A[Edit config files] --> B[apachectl configtest]
    B -->|exit 0| C[graceful reload]
    B -->|exit non-zero| D[Fix config, do not touch running server]
    C --> E[Check error log for child startup failures]
    E -->|clean| F[Reload landed]
    E -->|errors| G[New children failing - old config still serving]

Manual workflow:

# Gate the reload on a clean config test
apachectl configtest && apachectl graceful

For automation, apply the same pattern in whatever runs your deploys:

• Config management (Ansible, Puppet, Chef): validate the rendered config with httpd -t against the new files before notifying the reload handler. Most modules support a validate command on the template or file resource. A failed validation should fail the run, not skip the reload.
• systemd reload units: check what your distribution’s ExecReload actually runs. If it is a bare kill -USR1, wrap it or prefer apachectl graceful, and keep the habit of running configtest first yourself.
• Log rotation: rotation scripts that signal Apache should use graceful restart, not kill -HUP, which drops all connections. Rotation is not the moment to discover the config on disk is broken.

Verify reload success after the fact, not just config validity before it: check the error log for “resuming normal operations” and for child startup errors in the seconds after the reload. A reload is only done when the new generation is serving.

Signals to monitor

How Netdata helps

• Netdata collects Apache signals from mod_status at per-second granularity, so a reload that quietly failed shows up as a configuration that never changed its behavior, visible against the timeline of your deploy events.
• Uptime and restart tracking surfaces unexpected restarts and lets you line them up against config changes, which is how you catch a bad config that only detonated on the next restart.
• Error rate and worker utilization charts show the downstream signature of a stale config: you deploy a fix, the 5xx rate does not move, and that mismatch is the clue the reload never landed.
• Error log pattern monitoring catches the reload-time messages (“resuming normal operations”, syntax errors, child startup failures) that operators otherwise never grep for until an outage.
• Correlating deploy annotations with scoreboard state after a graceful restart reveals the nastier variant, where old-generation workers linger and new children fail, as a persistent shift in worker counts and memory after each reload.

Netdata’s Apache HTTP Server monitoring with Netdata brings these signals together with per-second metrics and ML anomaly detection.

Related guides

• Apache 500 Internal Server Error: modules, handlers, and misconfiguration
• Apache 502 Bad Gateway: a backend that returned an invalid response
• Apache 503 Service Unavailable: worker exhaustion versus proxy pool exhaustion
• Apache 504 Gateway Timeout: slow backends, ProxyTimeout, and worker pile-up
• Apache 5xx error rate: 500 vs 502 vs 503 vs 504 and what each one means
• Apache AH00558: Could not reliably determine the server’s fully qualified domain name
• Apache backend response time: telling ‘Apache is slow’ from ’the backend is slow’
• Apache balancer member in error state: reading balancer-manager and failover
• Apache BusyWorkers and IdleWorkers: reading worker utilization from mod_status
• Apache SSL certificate expired: the total, preventable HTTPS outage
• Apache CLOSE_WAIT and TIME_WAIT: connection leaks versus normal churn
• Apache CPU saturation: TLS handshakes, mod_deflate, mod_rewrite, and mod_security