eBPF OOMkill icon

eBPF OOMkill

eBPF OOMkill

Plugin: ebpf.plugin Module: oomkill

Overview

Monitor applications that reach out of memory.

Attach tracepoint to internal kernel functions.

This collector is only supported on the following platforms:

  • Linux

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

The plugin needs setuid because it loads data inside kernel. Netada sets necessary permission during installation time.

Default Behavior

Auto-Detection

The plugin checks kernel compilation flags (CONFIG_KPROBES, CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT), files inside debugfs, and presence of BTF files to decide which eBPF program will be attached.

Limits

The default configuration for this integration does not impose any limits on data collection.

Performance Impact

This thread will add overhead every time that an internal kernel function monitored by this thread is called.

Setup

Prerequisites

Compile kernel

Check if your kernel was compiled with necessary options (CONFIG_KPROBES, CONFIG_BPF, CONFIG_BPF_SYSCALL, CONFIG_BPF_JIT) in /proc/config.gz or inside /boot/config file. Some cited names can be different accoring preferences of Linux distributions. When you do not have options set, it is necessary to get the kernel source code from https://kernel.org or a kernel package from your distribution, this last is preferred. The kernel compilation has a well definedd pattern, but distributions can deliver their configuration files with different names.

Now follow steps:

  1. Copy the configuration file to /usr/src/linux/.config.
  2. Select the necessary options: make oldconfig
  3. Compile your kernel image: make bzImage
  4. Compile your modules: make modules
  5. Copy your new kernel image for boot loader directory
  6. Install the new modules: make modules_install
  7. Generate an initial ramdisk image (initrd) if it is necessary.
  8. Update your boot loader

Debug Filesystem

This thread needs to attach a tracepoint to monitor when a process schedule an exit event. To allow this specific feaure, it is necessary to mount debugfs (mount -t debugfs none /sys/kernel/debug/).

Configuration

File

The configuration file name for this integration is ebpf.d/oomkill.conf.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config ebpf.d/oomkill.conf

Options

Overwrite default configuration reducing number of I/O events

Examples

There are no configuration examples.

Metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Per cgroup

These metrics show cgroup/service that reached OOM.

This scope has no labels.

Metrics:

Metric Dimensions Unit
cgroup.oomkills cgroup name kills
services.oomkills a dimension per systemd service kills

Per apps

These metrics show cgroup/service that reached OOM.

Labels:

Label Description
app_group The name of the group defined in the configuration.

Metrics:

Metric Dimensions Unit
app.oomkill kills kills

Alerts

There are no alerts configured by default for this integration.

Troubleshooting

update every

ebpf load mode

lifetime

Get Netdata

Sign up for free

Want to see a demonstration of Netdata for multiple use cases?

Go to Live Demo