S3 Compatible Object Storage

S3 Compatible Object Storage

Plugin: go.d.plugin Module: s3check

Overview

Monitor S3-compatible object storage with active checks that write, read, list, and delete small probe objects and time their replication between sites.

The collector runs on the Netdata Agent as an ordinary S3 client, so every result includes what a real client experiences on that path: DNS, network, proxies, TLS, authentication, and the service itself. It reports the outcome and duration of each S3 operation, whether the content read back matches what was written, how long replication took compared with your objectives, how many probe objects still await cleanup, and whether new probes are paused because cleanup is backlogged.

One job runs one of three modes:

The lifecycle mode checks the life cycle of a probe object; it does not inspect S3 Lifecycle Management rules. Replication modes check one direction only: to validate a bidirectional setup, configure one job per direction.

Every job owns a private key space inside the configured prefix, <prefix><owner>/probe-..., where the owner segment is derived from this Agent and job. The collector writes, reads, lists, and deletes only keys it has recorded in that space. It never scans or touches other objects in the bucket, and Agents or jobs that share a bucket and prefix never interfere with each other.

Before a probe object is written, its key is recorded in a local journal under the Agent’s state directory. If a probe is interrupted by a restart, a network failure, or an ambiguous response, the recorded objects are deleted during later collections, a few per collection. Cleanup is best effort: it requires the same job name, mode, endpoints, buckets, prefix, and working credentials, and it cannot remove objects when the service or the permissions are gone. The number of objects awaiting cleanup is bounded. When that bound is reached, the collector keeps observing and cleaning up but stops creating new probe objects until a slot is free, and the mutation_backpressure chart shows the pause.

Replication modes poll the destination once per collection until the configured timeout. Each replication job has objectives, the propagation time you consider healthy, and timeouts, the point at which the probe is declared failed and its object is scheduled for cleanup. Delete propagation is measured as the moment the destination stops serving the object; how the storage system reclaims space behind that is not observed. In aws_replication mode cleanup deletes only the exact object versions and delete markers the probe created.

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

Grant each endpoint only the operations its role needs, scoped to the configured bucket and, for object operations, to the configured prefix:

The collector never writes to a destination; its delete permission is used only to remove the replicated copy of a probe object during cleanup. When assume_role is configured, the base credentials also need sts:AssumeRole on that role.

Default Behavior

Auto-Detection

There is no auto-detection. A job needs a mode, a source region and bucket, credentials or an AWS SDK credential source on the Agent host, and a destination for the replication modes.

Limits

Each job runs one probe at a time and cleans up a few pending objects per collection. The number of objects tracked for cleanup is a fixed internal bound; when it is reached, new probes pause until cleanup frees a slot. Probe objects are 4 KiB. Replication objectives must be at least one collection interval, because the destination is polled once per collection, and objectives and timeouts are capped at 24 hours.

Performance Impact

The impact on the Agent is negligible. On the storage side each collection issues a handful of small requests under the configured prefix: at most one new probe object with its reads, listing, and delete, plus a few cleanup deletes. Use a dedicated bucket or reserved prefix, credentials restricted to it, and a collection interval that matches how often you want to exercise the service.

Setup

You can configure the s3check collector in two ways:

Important

UI configuration requires paid Netdata Cloud plan.

Prerequisites

Choose a mode and prepare the buckets

The collector verifies the bucket contract of its mode before every collection and refuses to run when it is not met:

The destination key is always identical to the source key. Policies that rename or re-prefix objects are not supported. A dedicated bucket is the simplest way to meet these requirements, but any bucket that satisfies them works.

Every probe object is written with a conditional If-None-Match: * request so that a probe can never overwrite an existing key; the service must support conditional writes.

Reserve an object prefix

Pick a prefix, by default netdata-s3check/, and do not store your own data under it. The collector creates and deletes objects only under <prefix><owner>/, where the owner segment is unique to this Agent and job, so several Agents and jobs can safely share a prefix and even a bucket. Keys outside the collector’s own namespace are never listed, read, or deleted.

Create credentials with the minimum permissions

Each endpoint authenticates independently. Omit credentials to use the AWS SDK default credential chain on the Agent host (environment variables, shared config and profiles, instance or task roles), or provide static keys, preferably as go.d secret references such as ${env:NAME} rather than plaintext. Optionally add assume_role to switch to a role with those base credentials.

Grant only the operations listed in the permissions table of the Overview, scoped to the bucket and, for object operations, to the prefix. When assume_role is configured, the base credentials must also be allowed to call sts:AssumeRole on that role.

Run the job where your clients run

Results reflect the network path of the Agent that runs the job: DNS, routing, proxies, TLS, and authentication are all part of the measurement. Run each job on the Agent that best represents the clients you care about. Replication checks cover one direction; add a second job with source and destination swapped to check the reverse direction.

Configuration

Options

Set mode and configure only the matching mode_* object; the other two must be absent. Durations accept human-readable values such as 10s, 5m, 1h, or 1d.

Endpoint settings. Every source and destination object takes the same connection settings:

• endpoint, region, bucket, path_style: where the bucket lives. For AWS S3 leave endpoint empty and set path_style: no; for Ceph RGW and other S3-compatible services set the full URL and keep path-style addressing.
• credentials: static access keys. Omit the object to use the AWS SDK default credential chain of the Agent host. Prefer go.d secret references such as ${env:NAME} over plaintext values.
• assume_role: optional. The base credentials call STS AssumeRole and the returned temporary credentials are used for this endpoint only.
• timeout, proxy_url, and the tls_* options apply to every S3 request and to the STS AssumeRole request of that endpoint. An empty proxy_url honors the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment of the Agent.

Prefix. The collector only writes, reads, lists, and deletes keys under <prefix><owner>/, where the owner segment is derived from this Agent and job. Reserve the prefix for Netdata; several Agents and jobs may share it.

Objectives and timeouts (replication modes). An objective is the propagation time you consider healthy: exceeding it marks the write_visibility_objective or delete_visibility_objective chart as breached while the probe keeps waiting. A timeout is the hard limit: exceeding it fails the probe with reason visibility_timeout or delete_timeout and schedules the object for cleanup. Objectives must be at least update_every, because the destination is polled once per collection, timeouts must be at least their objective, and all four are capped at 24h.

mode

Select the mode first and configure only its mode_* object. A job whose mode object is missing, or that also sets another mode’s object, is rejected. The Overview compares the three modes.

mode_lifecycle

Each collection writes one 4 KiB probe object under the prefix, reads it back and verifies its content, confirms it appears in a listing, deletes it, and confirms it is gone. A successful collection leaves nothing behind.

The bucket must never have had versioning enabled; a suspended state is also rejected, because a plain delete must remove the object completely. Required permissions on the bucket: s3:GetBucketVersioning, s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject.

mode_ceph_multisite

Writes a probe object at the source zone, polls the destination zone once per collection until the same key is readable there with the same content, deletes it at the source, and polls until the destination no longer serves it. Write lag is measured from the source write, delete lag from the source delete.

Both buckets must be unversioned. The source needs s3:GetBucketVersioning, s3:PutObject, s3:GetObject, s3:DeleteObject; the destination needs s3:GetBucketVersioning, s3:GetObject, s3:DeleteObject. The collector never writes to the destination and deletes there only its own replicated copies during cleanup. Use one job per direction.

mode_aws_replication

Follows AWS S3 replication semantics: the source write creates an object version, the source delete creates a delete marker, and both are expected to replicate. The collector records the exact version and marker IDs on both sides and deletes only those during cleanup.

Both buckets need versioning enabled with MFA Delete off. The source bucket needs an enabled replication rule that covers the prefix without tag filters, targets the destination bucket, and has delete marker replication enabled; no other enabled rule may replicate the prefix to a different bucket. Source permissions: s3:GetBucketVersioning, s3:GetReplicationConfiguration, s3:PutObject, s3:GetObject, s3:GetObjectVersion, s3:ListBucketVersions, s3:DeleteObject, s3:DeleteObjectVersion. Destination permissions: s3:GetBucketVersioning, s3:GetObject, s3:ListBucketVersions, s3:DeleteObjectVersion. Use one job per direction.

via UI

Configure the s3check collector from the Netdata web interface:

1. Go to Nodes.
2. Select the node where you want the s3check data-collection job to run and click the :gear: (Configure this node). That node will run the data collection.
3. The Collectors → Jobs view opens by default.
4. In the Search box, type s3check (or scroll the list) to locate the s3check collector.
5. Click the + next to the s3check collector to add a new job.
6. Fill in the job fields, then click Test to verify the configuration and Submit to save.
   • Test runs the job with the provided settings and shows whether data can be collected.
   • If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File

The configuration file name for this integration is go.d/s3check.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
  - name: some_name1
  - name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/s3check.conf

Examples

Lifecycle check of an S3-compatible endpoint

Checks one Ceph RGW or other S3-compatible endpoint with static access keys read from environment variables. The bucket must never have had versioning enabled.

jobs:
  - name: s3_lifecycle
    mode: lifecycle
    mode_lifecycle:
      source:
        endpoint: https://s3.example.net
        region: us-east-1
        bucket: netdata-s3check
        credentials:
          access_key_id: ${env:NETDATA_S3CHECK_ACCESS_KEY_ID}
          secret_access_key: ${env:NETDATA_S3CHECK_SECRET_ACCESS_KEY}
        path_style: yes

Lifecycle check through a proxy with a private CA

The proxy and the CA bundle apply to this endpoint’s S3 requests (and to STS if assume_role were set). Credentials are omitted, so the AWS SDK default credential chain on the Agent host is used.

jobs:
  - name: s3_lifecycle_proxied
    mode: lifecycle
    mode_lifecycle:
      prefix: monitoring/netdata/
      source:
        endpoint: https://s3.internal.example.net
        region: us-east-1
        bucket: netdata-s3check
        proxy_url: http://proxy.example.net:3128
        tls_ca: /etc/ssl/private-ca.pem
        timeout: 15s

Ceph RGW multisite, one direction

Measures replication from zone A to zone B. Add a second job with source and destination swapped to check the reverse direction. Both buckets must be unversioned, and each zone uses its own keys.

jobs:
  - name: ceph_site_a_to_site_b
    mode: ceph_multisite
    mode_ceph_multisite:
      source:
        name: site-a
        endpoint: https://rgw-site-a.example.net
        region: us-east-1
        bucket: netdata-s3check
        credentials:
          access_key_id: ${env:NETDATA_S3CHECK_SITE_A_ACCESS_KEY_ID}
          secret_access_key: ${env:NETDATA_S3CHECK_SITE_A_SECRET_ACCESS_KEY}
      destination:
        name: site-b
        endpoint: https://rgw-site-b.example.net
        region: us-east-1
        bucket: netdata-s3check
        credentials:
          access_key_id: ${env:NETDATA_S3CHECK_SITE_B_ACCESS_KEY_ID}
          secret_access_key: ${env:NETDATA_S3CHECK_SITE_B_SECRET_ACCESS_KEY}
      write_objective: 15m
      write_timeout: 30m
      delete_objective: 5m
      delete_timeout: 15m

AWS S3 replication with a role per account

Checks replication from a bucket in one region to a bucket in another. Base credentials come from the AWS SDK default credential chain on the Agent host, for example an instance profile, and each endpoint then assumes its own role. Both buckets must be versioned and the source rule must replicate delete markers. path_style is disabled because AWS S3 uses virtual-hosted-style addressing.

jobs:
  - name: aws_replication
    mode: aws_replication
    mode_aws_replication:
      source:
        name: us-east-1
        region: us-east-1
        bucket: source-bucket
        path_style: no
        assume_role:
          role_arn: arn:aws:iam::[ACCOUNT]:role/netdata-s3-source
      destination:
        name: us-west-2
        region: us-west-2
        bucket: destination-bucket
        path_style: no
        assume_role:
          role_arn: arn:aws:iam::[ACCOUNT]:role/netdata-s3-destination

Troubleshooting

Diagnostics

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the s3check collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn’t working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that’s not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/
  

• Switch to the netdata user.

  sudo -u netdata -s
  

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m s3check
  

  To debug a specific job:

  ./go.d.plugin -d -m s3check -j jobName
  

Getting Logs

If you’re encountering problems with the s3check collector, follow these steps to retrieve logs and identify potential issues:

• Run the command specific to your system (systemd, non-systemd, or Docker container).
• Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep s3check

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector’s name:

grep s3check /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named “netdata” (replace if different), use this command:

docker logs netdata 2>&1 | grep s3check

Other Problems

The job fails its initial check

Before writing anything, the collector reads the bucket versioning state of every endpoint and, in aws_replication mode, the replication rules of the source bucket. A failure here means one of:

• the endpoint is not reachable from the Agent (DNS, routing, proxy, TLS, or region);
• the credentials are rejected or lack s3:GetBucketVersioning (or s3:GetReplicationConfiguration);
• the bucket does not meet the mode’s versioning contract: never versioned for lifecycle and ceph_multisite, versioning enabled with MFA Delete off for aws_replication;
• in aws_replication mode, no enabled rule covers the prefix and targets the destination bucket with delete marker replication, or another enabled rule replicates the prefix elsewhere.

Chart labels carry only a bounded reason; the exact provider error is in the collector log.

Objects remain pending cleanup or new probes are paused

After an interrupted probe the collector deletes the objects it recorded a few per collection. cleanup_pending_objects shows the backlog and mutation_backpressure turns active when the backlog reaches its bound, which pauses new probes but not observation or cleanup.

Cleanup needs the same job name, mode, prefix, endpoints, buckets, and working credentials that created the objects. If you changed the mode, prefix, endpoint, or bucket while objects were pending, the job refuses to start with an ownership mismatch error; if you renamed the job, the old objects are left behind. In both cases restore the previous configuration under the previous job name until the backlog drains, then change it. A cleanup reason means a probe object could not be confirmed gone, usually a permission problem; an ownership reason means the local state could not be locked or saved. The collector log has the details.

Replication probes stay in waiting or time out

A replication probe reports waiting until the destination serves the object (write) or stops serving it (delete), and fails with visibility_timeout or delete_timeout when the configured timeout passes. Check that the replication policy or zone sync covers the prefix, that the destination credentials can read the bucket, and that the policy delivers the object under exactly the same key. Persistent waits usually mean replication is slower than your objectives and timeouts assume; align them with what the replication setup promises.