Netdata’s database engine (dbengine) provides a sophisticated multi-tiered storage system designed for efficient long-term data retention while maintaining high granularity. This article explores the technical details of how Netdata handles metric storage, the advantages of its distributed architecture, and how to configure it for your specific needs.

Database Engine Architecture

Netdata’s database engine provides a sophisticated, efficient solution for long-term metric storage through:

• Intelligent multi-tiered storage architecture
• Efficient compression and caching mechanisms
• Flexible retention strategies
• Distributed deployment options

This design enables organizations to maintain detailed historical data while optimizing storage use and maintaining query performance.

Multi-Tier Storage Design

The dbengine implements a tiered storage approach where data is saved in multiple tiers on disk. Each tier retains data at a different resolution through an intelligent downsampling process:

1. Data Collection & Initial Storage

   • Metrics are collected at their original frequency
   • Initial storage occurs in Tier 0 at the native resolution of 1-second
   • Data is compressed immediately using efficient algorithms

2. Downsampling Process

   • Tier 1 creates per-minute points from 60 Tier 0 samples
   • Tier 2 creates per-hour points from 60 Tier 1 samples
   • Each tier maintains statistical accuracy through aggregation

3. Data Flow

Configuring Additional Tiers

The dbengine supports up to five storage tiers, giving you flexibility to create custom retention policies beyond the default three-tier setup. Each additional tier further downsamples the data from the previous tier, allowing for very long-term storage with minimal disk usage.

For example:

• Tier 0: per second (original resolution)
• Tier 1: per minute (60x downsample from Tier 0)
• Tier 2: per hour (60x downsample from Tier 1)
• Tier 3: per day (24x downsample from Tier 2)
• Tier 4: per week (7x downsample from Tier 3)

Additional tiers maintain excellent storage efficiency due to aggressive downsampling and compression. For example, storing 5 years of daily data points in Tier 3 often requires less storage than 1 year of hourly data in Tier 2.

Storage Efficiency Mechanisms

Compression and Data Formats

Each tier implements specific optimizations:

Memory Management

The database engine employs a sophisticated caching system:

1. Page Cache

   • Keeps frequently accessed metric data in memory
   • Dynamically adjusted based on system resources
   • Configurable through dbengine page cache size

2. Extent Cache

   • Manages compressed data blocks
   • Optimizes read operations for historical data
   • Configurable via dbengine extent cache size

[db]
    # Cache configuration example
    dbengine page cache size = 32
    dbengine extent cache size = 32

Distributed Architecture and Deployment Options

Netdata’s storage architecture is flexible enough to support both distributed and centralized deployment models, allowing you to choose the approach that best fits your infrastructure needs.

The dbengine provides complete control over data storage:

1. Data Location

   • All metrics stored within your infrastructure
   • No data stored in Netdata Cloud
   • Complete data sovereignty

2. Storage Organization

   /var/lib/netdata/
   ├── dbengine/
   │   ├── tier0/
   │   │   └── [metric data files]
   │   ├── tier1/
   │   │   └── [downsampled data]
   │   └── tier2/
   │       └── [long-term archives]
   

Standalone Node Deployment

In its simplest form, each Netdata node operates independently, maintaining its own metric history. This distributed approach offers several advantages: each node has complete control over its storage requirements, can scale its capacity independently, and maintains direct local access to its historical data.

This deployment model works particularly well for distributed infrastructures where each node needs autonomous operation. For instance, if you’re monitoring edge locations or remote sites with unreliable connectivity, standalone storage ensures each node maintains its complete metric history regardless of network conditions.

Each standalone node can be configured with its own retention policies and storage parameters through its local netdata.conf file:

[db]
    mode = dbengine
    storage tiers = 3
    dbengine tier 0 retention time = 14d
    dbengine tier 1 retention time = 3mo
    dbengine tier 2 retention time = 2y

Parent-Child Streaming Architecture

For organizations that prefer centralized historical data storage, Netdata supports a parent-child streaming architecture. In this model, child nodes stream their metrics to one or more parent nodes, which then maintain the long-term storage for the entire infrastructure.

The streaming architecture provides several benefits. Child nodes can be configured to run in RAM mode to minimize their resource usage, while parent nodes use the database engine for persistent storage. A typical parent node configuration might look like this:

[db]
    mode = dbengine
    storage tiers = 3
    # Increase retention for centralized storage
    dbengine tier 0 retention time = 30d
    dbengine tier 1 retention time = 6mo
    dbengine tier 2 retention time = 5y

The choice between these architectures isn’t mutually exclusive - many organizations implement a hybrid approach where some nodes maintain their own history while others stream to parent nodes. This flexibility allows you to adapt Netdata’s storage architecture to your specific monitoring requirements and infrastructure constraints.

Distributed Architecture Options

1. Standalone Node Storage

   • Each node maintains its own metric history
   • Independent scaling of storage capacity
   • Direct local access to historical data

2. Parent-Child Deployment

   • Child nodes stream metrics to parents
   • Parents maintain long-term storage
   • Efficient centralization of historical data

   

Retention Configuration

Retention Strategies

The database engine supports three distinct approaches:

1. Time-based Retention

   [db]
       dbengine tier 0 retention time = 14d
       dbengine tier 1 retention time = 3mo
       dbengine tier 2 retention time = 2y
   

2. Space-based Retention

   [db]
       dbengine tier 0 retention size = 1GiB
       dbengine tier 1 retention size = 1GiB
       dbengine tier 2 retention size = 1GiB
   

3. Combined Retention

   [db]
       # Tier 0 example with both limits
       dbengine tier 0 retention time = 14d
       dbengine tier 0 retention size = 1GiB
   

RAM mode for short term storage

While the database engine is Netdata’s primary storage mechanism, Netdata also provides a RAM-only storage mode for specific use cases. This alternative mode stores all metric data directly in memory without touching the disk, offering a different set of tradeoffs compared to the database engine.

RAM mode is particularly useful in parent-child streaming setups. In these architectures, child nodes can run in RAM mode to minimize resource usage and storage overhead, while streaming their data in real-time to a parent node that maintains the long-term storage using the database engine.

The memory footprint in RAM mode is predictable and efficient. Each metric sample requires exactly 4 bytes of memory, and Netdata uses memory-mapped files (mmap()) for the ring buffers, which can be incremented in steps of 1024 samples (4KiB). When running on Linux, RAM mode can take advantage of the kernel memory deduplication (KSM) to optimize memory usage further.

You can enable RAM mode by modifying your netdata.conf:

[db]
    mode = ram
    # Optional: customize retention in seconds
    retention = 3600

Remember that data stored in RAM mode exists only in memory and will be lost when Netdata restarts. While this might seem like a limitation, it’s actually ideal for child nodes in a streaming setup, as the parent node maintains the persistent storage and historical data.

Operational Considerations

Planning Your Storage Strategy

When implementing Netdata’s long-term storage, careful planning ensures optimal performance and efficient resource utilization. Let’s explore the key considerations for capacity planning and ongoing maintenance.

Estimating Storage Requirements

Storage requirements for Netdata’s database engine depend on several factors. First, consider your metrics volume - how many metrics you collect and their collection frequency. With Netdata’s efficient compression, each metric typically requires about 0.6 bytes per sample in Tier 0, scaling to about 6 bytes in Tier 1 and 18 bytes in Tier 2.

For example, if you’re collecting 1,000 metrics every second, you can estimate your daily storage needs:

Tier 0: 1000 metrics * 86400 seconds * 0.6 bytes ≈ 50MB per day
Tier 1: 1000 metrics * 1440 minutes * 6 bytes ≈ 8MB per day
Tier 2: 1000 metrics * 24 hours * 18 bytes ≈ 0.4MB per day

Remember to account for future growth in both metrics and retention requirements. It’s generally recommended to allocate at least 50% extra capacity for unexpected increases in data collection.

Performance Optimization

Optimizing Netdata’s database performance involves careful configuration of memory resources. The two main components to consider are the page cache and extent cache sizes. The page cache keeps frequently accessed metric data in memory, while the extent cache maintains compressed data blocks.

For optimal performance, monitor your system’s memory usage and adjust these cache sizes accordingly. A good starting point is to allocate 32MB to each cache:

Watch your system’s disk I/O patterns during normal operation. If you notice high disk activity, you might need to increase cache sizes or adjust your retention strategy to better match your available resources.

Ongoing Monitoring

Netdata itself provides detailed metrics about its database engine performance. Regular monitoring of these metrics helps you maintain optimal performance and catch potential issues before they impact your monitoring capabilities.

Frequently Asked Questions

Q: How does the tiering system handle gaps in data collection?
A: The database engine maintains temporal consistency across tiers, properly handling collection interruptions while preserving statistical accuracy in higher tiers.

Q: Can I modify tier retention periods dynamically?
A: Yes, retention settings can be modified at runtime through netdata.conf. Changes take effect for new data while preserving existing historical data within the new limits.

Q: How does parent-child streaming affect storage requirements?
A: Parent nodes need sufficient storage for aggregated data from all child nodes. Child nodes in streaming configurations can use RAM mode to minimize local storage.

Q: What happens if a tier reaches its size limit but hasn’t hit its time limit?
A: The oldest data in that tier will be removed to maintain the size limit, even if it hasn’t reached the time limit.

Q: How does the database engine handle system restarts?
A: The database engine is designed for durability, maintaining data consistency across system restarts. The page cache is rebuilt as needed when Netdata restarts.

Q: Can I access historical data programmatically?
A: Yes, historical data is accessible through Netdata’s API endpoints, allowing programmatic access while respecting the tiering structure.