$ guides / logstash / logstash-dateparsefailure ▌

Operations Guides

Logstash _dateparsefailure: timestamp formats that stop parsing

The _dateparsefailure tag appears when Logstash’s date filter cannot parse a timestamp field against any of its configured match patterns. The event still flows through the pipeline and reaches its output. @timestamp stays at ingestion time instead of being set to the event’s actual time.

The result: time-based dashboards show gaps or misplaced data points, index lifecycle management operates on ingestion time so events land in the wrong time bucket or expire at the wrong time, and the issue can persist for days because throughput metrics stay green. The date filter tries each format string in the match array sequentially. The first successful parse wins. If none succeed, the filter appends its tag_on_failure tag (default _dateparsefailure) and leaves @timestamp unchanged.

The most common trigger is a source system changing its timestamp format after an upgrade or reconfiguration. Other causes: locale mismatches for month and day names, timezone handling errors, Daylight Saving Time edge cases, and format specifier differences between Joda-Time and java.time parser modes.

What this means

A _dateparsefailure tag is a silent correctness failure. Pipeline throughput looks normal because events are not dropped, filtered, or delayed. The events-in, events-out, and queue metrics all stay green. The only evidence is the tag itself and the gap between @timestamp and the source timestamp.

There is no direct metric in the Logstash stats API for the _dateparsefailure tag specifically. Detection relies on querying the downstream destination for the tag. The per-plugin stats may expose a failures counter for the date filter, but downstream inspection is the reliable method.

Blast radius depends on how many events carry the bad format. If a single high-volume source changes its timestamp format, a large percentage of events in the destination index may carry wrong timestamps without any throughput metric moving.

flowchart TD
    A["_dateparsefailure on events"] --> B{"Recent config deploy?"}
    B -->|Yes| C["Check date filter match patterns"]
    B -->|No| D["Source changed timestamp format"]
    C --> E{"All events or subset?"}
    E -->|All| F["Pattern syntax or locale issue"]
    E -->|Subset| G["New timestamp variant from source"]
    D --> H["Sample failed event timestamp field"]
    H --> I["Compare against configured match patterns"]
    I --> J["Add missing format to match array"]
    F --> K{"Contains month or day names?"}
    K -->|Yes| L["Check locale parameter"]
    K -->|No| M["Check timezone or DST gap"]

Common causes

Cause	What it looks like	First thing to check
Source timestamp format changed	Sudden spike in tag rate from one source, often after a deployment on the source side	Sample the raw timestamp field from a failed event
Locale mismatch	Patterns with `MMM` or `EEE` (month or day names) fail; JVM locale is non-English	Run `locale` on the Logstash host and check the `locale` parameter in the date filter
Timezone not specified	Timestamps without embedded offset fail or parse to wrong time	Check whether the date filter has a `timezone` parameter set
DST gap timestamp	Failures spike once or twice per year at DST transition in affected timezone	Check if failed timestamps fall in the spring-forward gap (for example, 02:00 in America/New_York on the transition date)
Joda-Time vs java.time specifier mismatch	Custom patterns fail after switching to `precision => "ns"`	Check which parser mode is active and verify format specifiers against the correct library
Fractional seconds beyond SSS	Timestamps with more than 3 fractional digits fail in default (Joda-Time) mode	Check if the source emits nanosecond or microsecond precision

Quick checks

# Count _dateparsefailure tags in Elasticsearch (add auth as needed)
curl -s "localhost:9200/<index>/_count?q=tags:_dateparsefailure"

# Check the date filter configuration
grep -A 20 'date\s*{' /etc/logstash/conf.d/*.conf

# Inspect per-filter stats for the date filter
curl -sS http://127.0.0.1:9600/_node/stats/pipelines?pretty | \
  python3 -c "
import sys,json
pipelines = json.load(sys.stdin)['pipelines']
for pname, pdata in pipelines.items():
    for f in pdata.get('plugins',{}).get('filters',[]):
        if f.get('name') == 'date':
            evts = f.get('events', {})
            print(f'Pipeline: {pname}, Filter ID: {f.get(\"id\")}')
            print(f'  failures: {evts.get(\"failures\", \"not reported\")}')
            print(f'  in: {evts.get(\"in\", \"N/A\")}, out: {evts.get(\"out\", \"N/A\")}')
"

# Check Logstash logs for date parse errors
grep -i 'dateparsefailure\|date.*filter\|parsing.*date' /var/log/logstash/logstash-plain.log | tail -n 100

# Check the JVM default locale
locale

# Sample a failed event to see its raw timestamp
curl -s "localhost:9200/<index>/_search?size=1&q=tags:_dateparsefailure" | \
  python3 -c "import sys,json; h=json.load(sys.stdin)['hits']['hits']; print(json.dumps(h[0]['_source'], indent=2)) if h else print('No results')"

How to diagnose it

Confirm the tag is present and quantify the scope. Query the downstream for _dateparsefailure tag count and compare it to total event count. A sudden spike after a known source-side deployment points to a format change. A steady low rate may indicate a long-standing mismatch that was never caught.
Sample failed events to extract the raw timestamp. Pull a few events with the tag and look at the source timestamp field referenced in the date filter’s match parameter. Write down the exact format, including separators, timezone notation, and fractional second precision.
Compare against configured patterns. Open the date filter configuration and compare the raw timestamp against each format string in the match array. Common mismatches: separator changes (space to T, slash to hyphen), timezone format changes (offset to abbreviation or vice versa), precision changes (added or removed fractional seconds), or date ordering changes.
Check locale and timezone. If the timestamp contains month names (Jan, Feb) or day names, verify the locale parameter. If the JVM default locale is non-English and the filter has no explicit locale, month name parsing may fail. If timestamps lack a timezone offset, verify the timezone parameter is set.
Test the fix before deploying. Use a Logstash config test to validate syntax: logstash -t -f /etc/logstash/conf.d/<file>.conf. This checks configuration syntax without starting the pipeline. If possible, replay a sample of the actual failed events through the updated filter locally.

Metrics and signals to monitor

Signal	Why it matters	Warning sign
`_dateparsefailure` tag rate at destination	Direct evidence of date parse failures; no direct API metric exists for this tag	Any sudden increase above baseline, or any sustained non-zero rate on critical streams
Date filter failures (per-plugin stats)	Filter-level failure count from the stats API	Counter delta increasing per interval
Pipeline events-out rate	Confirms throughput is unaffected (expected for this failure mode)	Should remain normal; a drop would indicate a different problem
Config reload state (`reloads.failures`)	A failed reload may leave an old date filter running while the team believes the fix was deployed	Non-zero failures after a config deploy
Source timestamp format distribution	Detects format drift before it becomes a parse failure	New format variants appearing in the source data

Fixes

Source timestamp format changed

Add the new format to the date filter’s match array. The filter tries formats in order, so keep existing formats and append the new one:

date {
  match => ["log_timestamp", "MMM dd yyyy HH:mm:ss", "yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
}

If you control the source, standardize on ISO 8601 and use the ISO8601 literal, which handles the format robustly.

Locale mismatch

Set the locale parameter explicitly when timestamps contain month or day names:

date {
  match => ["log_timestamp", "MMM dd HH:mm:ss"]
  locale => "en"
}

Timezone not specified

If timestamps lack an embedded timezone offset, set the timezone parameter to a canonical IANA timezone ID:

date {
  match => ["log_timestamp", "yyyy-MM-dd HH:mm:ss"]
  timezone => "America/New_York"
}

For sources that emit UTC timestamps without a trailing Z, set timezone => "UTC".

DST gap timestamps

The underlying Joda-Time library rejects timestamps that fall into a Daylight Saving Time gap. For example, 2019-03-10 02:00:00 in America/New_York does not exist because clocks spring forward from 01:59:59 to 03:00:00. If the source data is actually in UTC, set timezone => "UTC" to avoid the gap entirely.

Joda-Time vs java.time specifier mismatch

Custom format patterns use Joda-Time by default. Setting precision => "ns" switches to java.time for nanosecond precision, but the two libraries have different format specifiers:

Timezone ID: ZZZ in Joda-Time, VV in java.time
Year: y works as year-of-era in Joda-Time.

If you recently switched to precision => "ns" and parsing broke, verify that your format specifiers are valid for java.time.

Fractional seconds precision

The Joda-Time parser matches exactly the number of fractional second digits specified by SSS (3 digits). If the source emits more digits, the extra characters cause the remaining pattern to fail. For sources emitting nanosecond or microsecond precision, switch to java.time with precision => "ns" and adjust the pattern accordingly.

Timezone names cannot be parsed

The Joda-Time parser does not reliably parse timezone names like EST or PDT using the z specifier. Use Z, ZZ, or ZZZ for numeric offsets and timezone IDs instead. If the source emits timezone abbreviations, fix the source or use a mutate filter to convert the abbreviation to a numeric offset before the date filter.

Prevention

Alert on _dateparsefailure tag rate for time-critical streams. This is not exposed as a built-in Logstash API metric. Query the downstream destination on a schedule, or add a metrics filter in the pipeline to count tagged events.
List multiple formats in the match array. Keep old formats alongside new ones. The first successful parse wins, so extra entries carry negligible overhead.
Set locale and timezone explicitly. Do not rely on JVM defaults, which may differ between hosts or change after a JVM upgrade.
Coordinate with application teams on timestamp format changes. The most common trigger is a source-side deployment that changes the log format. Early communication lets you update the date filter proactively.
Test config changes against real data samples. Run logstash -t -f <config> for syntax validation, and if possible, replay a sample of actual events through the updated filter before deploying.

How Netdata helps

Per-second pipeline metrics: Netdata collects Logstash API metrics at per-second resolution, making the onset of date filter failures visible immediately rather than after a multi-minute polling delay.
Config reload correlation: If a config reload coincides with a change in event processing patterns, the reload metrics (reloads.successes, reloads.failures) help distinguish a deployment-induced problem from a source-side format change.
Per-pipeline visibility: In multi-pipeline deployments, per-pipeline metrics isolate which pipeline is affected, preventing aggregate metrics from masking a problem in one pipeline.
Anomaly detection on event rates: ML-based anomaly advisors can flag subtle shifts in event processing patterns that correlate with the start of parse failures, even when throughput technically stays constant.