MySQL Operations Guides on Netdata

How MySQL actually works in production: a mental model for operators

Mon, 01 Jan 0001 00:00:00 +0000

How MySQL actually works in production: a mental model for operators

MySQL is a connection-oriented request processor with a pluggable storage engine. In production, that engine is almost always InnoDB. Operating MySQL at scale means managing memory pressure, write-ahead log capacity, version-chain traversal, and fsync latency. The optimizer matters, but the stalls that wake you up start inside InnoDB.

This guide maps the subsystems that fail in production. It assumes you know how to run SHOW PROCESSLIST and need to move from symptoms to root cause: why the server stalls, why replication drifts, or why a fast query suddenly saturates disk I/O.

MySQL Aborted_connects and Aborted_clients climbing: diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Aborted_connects and Aborted_clients climbing: diagnosis

You notice Aborted_connects or Aborted_clients climbing on a production MySQL instance. Because these are cumulative counters, a steady upward slope means something is actively failing or dropping connections. In a busy system, a rising Aborted_connects rate can trigger host blocking via max_connect_errors, suddenly preventing legitimate clients from connecting. A rising Aborted_clients rate usually shows up as application-side exceptions about closed connections, forcing retries that can cascade into connection exhaustion. The two counters track completely different failure modes: Aborted_connects counts connection attempts that never finished authentication; Aborted_clients counts connections that authenticated but then died unexpectedly. Treating them as the same metric leads to wrong fixes. Restarting the network stack will not fix a credential rotation bug, and increasing max_connections will not fix a client killed by wait_timeout.

MySQL adaptive hash index latch contention: high CPU, low throughput

Mon, 01 Jan 0001 00:00:00 +0000

MySQL adaptive hash index latch contention: high CPU, low throughput

Burning CPU while barely answering queries: OS user time near 100%, application latency spiking, and Questions flat or falling. Threads_connected is high but Threads_running stays low, and disk I/O is quiet. This pattern matches an Adaptive Hash Index (AHI) latch storm.

InnoDB builds an in-memory hash index over frequently accessed B-tree pages to speed lookups. Since MySQL 5.7, the latch is partitioned into innodb_adaptive_hash_index_parts (default 8). Under high concurrency, especially point lookups on a hot index, threads can collide on the same partition latch and spin instead of executing. CPU saturates while throughput collapses.

MySQL authentication failure spike: brute force vs broken credential rotation

Mon, 01 Jan 0001 00:00:00 +0000

MySQL authentication failure spike: brute force vs broken credential rotation

You get paged because Aborted_connects is climbing. The error log shows a wall of authentication failures, application dashboards are yellow, and someone in security is asking if this is an attack. Before you block IP addresses or rotate passwords, you need to know which failure mode you are dealing with. External brute force and internal broken credential rotation produce the same MySQL status counters, but the fixes are opposite. One requires closing the network window and flushing blocked hosts. The other requires finding the application instance that missed the new secret.

MySQL binary logs filling the disk: expiry, lagging replicas, and purge

Mon, 01 Jan 0001 00:00:00 +0000

MySQL binary logs filling the disk: expiry, lagging replicas, and purge

You get a disk-full alert on the MySQL primary. df -h shows the data partition at 95%, and du points to /var/lib/mysql/binlog.* consuming hundreds of gigabytes. Writes are about to fail.

Binary logs are append-only. MySQL rotates to a new file at max_binlog_size (default 1 GB), but rotation does not delete old files. Deletion only happens via automatic expiry or PURGE BINARY LOGS. MySQL refuses to delete any binlog file that a connected replica has not yet consumed. In MySQL 5.7, the default expiry is never (expire_logs_days = 0). In 8.0, the default is 30 days, but automatic purge cannot remove files that a connected replica still needs. A replica lagging past the expiry window therefore causes unbounded growth.

MySQL connection exhaustion: detection, diagnosis, and prevention

Mon, 01 Jan 0001 00:00:00 +0000

MySQL connection exhaustion: detection, diagnosis, and prevention

ERROR 1040 (HY000): Too many connections. Health checks fail. Users cannot sign in. If you configured an admin account correctly, a break-glass session may still get through on the reserved slot, and SELECT 1 returns. MySQL is running, but the connection pool is a wall.

This is a hard cliff. One moment queries flow; the next, every new TCP handshake to port 3306 is rejected. The server does not queue connections. Whether the root cause is a connection leak, a retry storm, or genuine overload from slow queries, the symptom is the same: Threads_connected has reached max_connections.

MySQL Created_tmp_disk_tables: temp tables spilling to disk

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Created_tmp_disk_tables: temp tables spilling to disk

Queries that GROUP BY, ORDER BY, DISTINCT, or materialize derived results build internal temporary tables. When those tables exceed memory limits, MySQL spills them to disk. The status counter Created_tmp_disk_tables tracks this, and its ratio to Created_tmp_tables is the signal to watch during a latency incident.

A low ratio is normal. A high ratio means queries pay a random-disk penalty for intermediate results. You see that as rising query latency, increased disk I/O on the data directory, and connection pile-up. This guide covers how to read the ratio, find the offending patterns, and fix them without just adding RAM.

MySQL ERROR 1040 (HY000): Too many connections - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1040 (HY000): Too many connections - causes and fixes

Once Threads_connected reaches max_connections, MySQL returns ERROR 1040 (HY000) to every new connection attempt before authentication. The server is not down; it is full.

This error is usually a symptom. Connections may be leaking from the application, a cache stampede may have flooded the pool, or slow queries may be holding slots open longer than expected. MySQL reserves one extra connection for users with CONNECTION_ADMIN (or the deprecated SUPER privilege). If that slot is free, an operator can still connect. If an app user has taken it, you may need to restart.

MySQL ERROR 1045 (28000): Access denied for user - diagnosis

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1045 (28000): Access denied for user - diagnosis

When you see ERROR 1045 (28000): Access denied for user 'app'@'10.0.0.5' (using password: YES), the connection never reached the query parser. MySQL rejected it during authentication, incremented Aborted_connects, and returned SQLSTATE 28000. The error always includes the effective user and host as seen by the server. This is your first and most important clue.

This error has four root causes in production: wrong credentials, user@host mismatch, authentication plugin incompatibility, and host-level blocking from accumulated connection errors. A brute-force probe and a misconfigured deploy look identical from the server side, so diagnosis hinges on correlating the error pattern with connection metrics and the host cache.

MySQL ERROR 1205: Lock wait timeout exceeded; try restarting transaction

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1205: Lock wait timeout exceeded; try restarting transaction

ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction appears when one transaction holds an InnoDB row lock so long that another transaction exhausts innodb_lock_wait_timeout (default 50 seconds). Unlike a deadlock, which InnoDB resolves automatically, a timeout means a blocker is still active. The root cause is almost always a long-running transaction or hot row contention. Find it before it cascades into a wider outage.

MySQL ERROR 1213: Deadlock found when trying to get lock; try restarting transaction

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1213: Deadlock found when trying to get lock; try restarting transaction

Your application logs show ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction. One transaction was rolled back; the other completed normally. InnoDB broke a circular lock wait by selecting the cheaper transaction as the victim. A few deadlocks per hour in high-concurrency OLTP is normal. A sustained storm is not: it means transactions are colliding under load.

MySQL ERROR 1290: --read-only option so it cannot execute this statement

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 1290: –read-only option so it cannot execute this statement

A write that fails with ERROR 1290 (HY000): The MySQL server is running with the --read-only option so it cannot execute this statement is a routing or topology problem, not a query problem. The server refuses the statement because read_only or super_read_only is enabled. You are likely sending writes to a replica, a recently promoted primary that never cleared its read-only flag, or a node correctly enforcing super_read_only. Confirm the instance role and fix the write path.

MySQL ERROR 2006/2013: MySQL server has gone away -- causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 2006/2013: MySQL server has gone away – causes and fixes

ERROR 2006 (CR_SERVER_GONE_ERROR) and ERROR 2013 (CR_SERVER_LOST) produce the messages “MySQL server has gone away” and “Lost connection during query.” They often appear together, but they indicate different failures. ERROR 2006 means the server closed a connection the client believed was valid. ERROR 2013 means the connection dropped while a query was executing or results were transferring.

Treating them identically leads to incorrect fixes. Raising max_connections does not stop idle timeouts. Tuning timeouts does not fix an OOM kill. Before changing configuration, determine whether the MySQL process restarted. A stable process points to timeout or packet mismatches. A restarted process points to a crash or memory exhaustion.

MySQL ERROR 24 (HY000): Too many open files -- raising open_files_limit

Mon, 01 Jan 0001 00:00:00 +0000

MySQL ERROR 24 (HY000): Too many open files – raising open_files_limit

ERROR 24 (HY000): Too many open files means the OS refused a file descriptor request because the mysqld process hit its hard ceiling. Clients may see Can't create/write to file with errno: 24. Tables that were accessible moments ago fail to open, and new connections may be rejected even though Threads_connected is well below max_connections.

MySQL holds file descriptors for cached tables, client connections, binary logs, relay logs, and on-disk temporary tables. When combined demand exceeds the OS-enforced limit, the server returns errno 24. Unlike connection exhaustion, which is gated by max_connections, file descriptor exhaustion is gated by open_files_limit and the OS limits that enforce it. Raising open_files_limit in my.cnf is not enough if systemd, PAM, or the shell ulimit blocks the request.

MySQL FLUSH TABLES WITH READ LOCK stall: backups that freeze the server

Mon, 01 Jan 0001 00:00:00 +0000

MySQL FLUSH TABLES WITH READ LOCK stall: backups that freeze the server

Your application suddenly cannot write. Threads_connected climbs toward max_connections, but Questions flatlines. The last change was a backup job that started ten minutes ago. The culprit is almost always FLUSH TABLES WITH READ LOCK (FTWRL), and the damage is caused not by the lock itself but by what happens while the server waits to acquire it.

FTWRL is triggered by mysqldump --master-data, mysqldump --lock-tables, and similar tools that need a consistent logical backup. It attempts to close all open tables and acquire a global read lock. While it waits for a long-running query to finish, new writes are already blocked. The result is a whole-server freeze: the query rate drops to zero, connections pile up, and the application sees cascading timeouts.

MySQL full table scans: Handler_read_rnd_next and the missing index

Mon, 01 Jan 0001 00:00:00 +0000

MySQL full table scans: Handler_read_rnd_next and the missing index

You get paged because query latency is spiking. Slow_queries is climbing. Handler_read_rnd_next has jumped five-fold over its baseline and keeps rising. Connections, CPU, and buffer pool hit ratio look fine. The culprit is usually a new full table scan after a deploy, schema change, or query pattern shift.

Handler_read_rnd_next increments when the storage engine reads the next row during a table scan or sorted retrieval. In a healthy OLTP system, Handler_read_key dominates and Handler_read_rnd_next stays flat relative to query volume. When the ratio inverts, you are looking at a query plan regression, a missing index, or an optimizer decision gone wrong.

MySQL gap locks and next-key locks: surprising deadlocks under REPEATABLE READ

Mon, 01 Jan 0001 00:00:00 +0000

MySQL gap locks and next-key locks: surprising deadlocks under REPEATABLE READ

MySQL deadlocks in production frequently involve transactions that modify apparently unrelated rows. Two concurrent UPDATEs with narrow WHERE clauses collide, or an UPDATE blocks every INSERT into the table. The cause is usually REPEATABLE READ combined with InnoDB next-key locking and a missing or non-unique index.

Under REPEATABLE READ, the default in MySQL, InnoDB does not lock only matching rows. To prevent phantom reads, it locks the gaps between index entries. When no suitable index exists, or when the optimizer scans a non-unique index, a targeted UPDATE escalates into a range lock covering far more of the table than the predicate suggests.

MySQL Got error 28 from storage engine / No space left on device — recovery

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Got error 28 from storage engine / No space left on device — recovery

When Got error 28 from storage engine appears, the underlying filesystem is at 100%. MySQL needs writable space for InnoDB data files, redo logs, binary logs, temporary tables, relay logs, and slow query logs. Writes fail immediately. Affected threads enter a retry loop, logging warnings until an operator frees space or kills the operation.

The most dangerous scenario is the redo log filling up. If checkpoint age approaches capacity, InnoDB forces aggressive synchronous flushing. Under sustained pressure this can trigger an unclean shutdown and crash recovery on restart. Replicas are similarly vulnerable: a full relay log partition stops the I/O thread and lets lag grow unboundedly, risking binlog expiry on the source before catch-up.

MySQL GTID errant transactions: detecting replication divergence

Mon, 01 Jan 0001 00:00:00 +0000

MySQL GTID errant transactions: detecting replication divergence

Before a planned failover, or after an orchestrator aborts with a GTID consistency error, replication can appear healthy: SHOW REPLICA STATUS reports both threads running, Seconds_Behind_Source is near zero, and the error log is quiet. Yet comparing GTID sets between source and replica reveals mismatching numbers. Errant transactions are GTIDs present in a replica’s gtid_executed set that the source never generated.

Ordinary lag resolves as the replica catches up. Errant transactions represent true divergence. Promoting a replica with extra GTIDs propagates those transactions into the new source and downstream replicas, creating split-brain that is expensive to reverse. This guide covers distinguishing benign lag from dangerous divergence, pinpointing offending GTIDs, and deciding between empty-transaction injection and a full rebuild.

MySQL InnoDB buffer pool hit ratio collapse: the cliff edge

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB buffer pool hit ratio collapse: the cliff edge

Your OLTP queries were running in single-digit milliseconds. Now every query is taking seconds, the disk subsystem is saturated, Threads_running is climbing toward max_connections, and the buffer pool hit ratio, which sat at 99.9% for months, just fell through 95% and keeps dropping.

This is the InnoDB buffer pool cliff edge. When the working set exceeds the buffer pool, pages are evicted before they can be reused. Every miss becomes a physical disk read. The degradation is non-linear: 99.9% to 99% is a slow bleed, 99% to 95% is rapid, and below 95% disk saturation, uniform latency inflation, and connection exhaustion turn a capacity problem into an availability incident.

MySQL InnoDB checkpoint age: the redo log capacity signal nobody watches

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB checkpoint age: the redo log capacity signal nobody watches

Checkpoint age is the distance between the current InnoDB log sequence number (LSN) and the last checkpoint LSN. It measures how much of the circular redo log is occupied by changes not yet flushed to data files. When this age approaches redo log capacity, InnoDB escalates flushing aggressiveness until it runs out of options and stalls all writes.

Standard MySQL does not expose Innodb_checkpoint_age as a status variable. Depending on version, you must parse SHOW ENGINE INNODB STATUS, enable a disabled-by-default INNODB_METRICS counter, or compute the delta from MySQL 8.0.30+ redo status variables. MySQL 8.0.30 replaced innodb_log_file_size and innodb_log_files_in_group with innodb_redo_log_capacity.

MySQL InnoDB history list length growing: purge lag explained

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB history list length growing: purge lag explained

If query latency climbs across every table and no single query stands out, check the InnoDB history list length. It tracks unpurged undo records: MVCC debt. When purge falls behind, every consistent read walks longer version chains, and the slowdown is global. Unlike a slow query, this debt is invisible to conventional query analysis.

In MySQL 8.0, the programmatic source for this metric is trx_rseg_history_len in information_schema.INNODB_METRICS. Despite being labeled status_counter in the metrics table, trx_rseg_history_len behaves as a gauge: it rises when purge lags and falls when purge catches up.

MySQL InnoDB redo log checkpoint stall: when all writes freeze

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB redo log checkpoint stall: when all writes freeze

InnoDB redo log checkpoint stalls freeze all writes synchronously when checkpoint age reaches the end of the circular redo log. INSERTs, UPDATEs, and DELETEs that complete in milliseconds hang for seconds. Threads_running climbs while the Questions rate collapses. Read queries may still return initially, but soon every connection waits for a write lock or commit acknowledgment. Then throughput recovers just as abruptly. This is not a disk failure, deadlock, or runaway query. It is MySQL’s write path running out of runway.

MySQL InnoDB row lock contention: finding who blocks whom

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB row lock contention: finding who blocks whom

Queries that normally finish in milliseconds take seconds. Threads_running climbs while Questions stalls. SHOW PROCESSLIST shows active threads, yet the database is frozen. That is the shape of InnoDB row lock contention: transactions are waiting to release row-level locks, and the queue is growing.

Row lock contention differs from a metadata lock cascade. Metadata locks block DDL and DML at the table level and live in performance_schema.metadata_locks. Row locks are held by open InnoDB transactions and block at the row, gap, or next-key level. Rising Innodb_row_lock_current_waits alongside rising Threads_running signals a live contention crisis. The goal is to identify the blocker, the waiter, and the lock footprint so you can break the chain without guessing.

MySQL innodb_buffer_pool_size tuning: 60-80% of RAM and when that breaks

Mon, 01 Jan 0001 00:00:00 +0000

MySQL innodb_buffer_pool_size tuning: 60-80% of RAM and when that breaks

The 60-80% rule works for a bare-metal host running only MySQL. In containers, shared hardware, and high-connection-count environments, it is a hazard. Size the pool against aligned allocation, cgroup limits, per-connection memory, and OS headroom. Do not size it as a percentage of total RAM.

What the buffer pool costs

InnoDB caches data and index pages in a fixed-size pool. Every read and write touches it. When the working set fits entirely in memory, queries avoid disk. When it does not, InnoDB evicts pages and disk reads dominate latency.

MySQL Innodb_buffer_pool_wait_free > 0: buffer pool memory pressure

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Innodb_buffer_pool_wait_free > 0: buffer pool memory pressure

Innodb_buffer_pool_wait_free increments when InnoDB must synchronously flush dirty pages to make room for new reads. A sustained nonzero rate means queries are waiting on disk writes before they can proceed. Unlike the buffer pool hit ratio, which can stay above 99% while the system stalls, wait_free confirms the buffer pool is operating at its limit.

This is the Buffer Pool Cliff pattern: once the working set exceeds available clean pages, performance degrades non-linearly, disk I/O saturates, and threads pile up.

MySQL innodb_deadlock_detect=OFF: when deadlock detection becomes the bottleneck

Mon, 01 Jan 0001 00:00:00 +0000

MySQL innodb_deadlock_detect=OFF: when deadlock detection becomes the bottleneck

At extreme concurrency, InnoDB’s deadlock detector can become the bottleneck. It traverses the wait-for graph on every lock enqueue. That cost is usually negligible until thousands of concurrent writing transactions hit a small set of hot rows. When traversal itself saturates CPU, throughput collapses even though disks are idle and the buffer pool is warm.

innodb_deadlock_detect=OFF removes that traversal. Instead of instant deadlock detection, InnoDB relies exclusively on innodb_lock_wait_timeout to break cycles. The blocked transaction waits, holding its locks, until the timeout expires. This is not a generic performance tuning knob. It is a narrow, high-risk tradeoff that converts instant rollback into delayed timeout, and it is appropriate only when you have confirmed through testing that deadlock detection is the bottleneck.

MySQL Innodb_log_waits > 0: the log buffer is too small (not a checkpoint stall)

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Innodb_log_waits > 0: the log buffer is too small (not a checkpoint stall)

InnoDB increments Innodb_log_waits every time a thread waits because the log buffer is full. A sustained nonzero rate means the buffer cannot absorb your write burst, or the storage layer cannot drain it fast enough.

This is not a checkpoint stall. Checkpoint stalls happen when the on-disk redo log fills and InnoDB forces synchronous dirty-page flushing. Innodb_log_waits measures in-memory log buffer pressure only. Confusing the two leads to tuning redo log file size while the real problem persists.

MySQL innodb_redo_log_capacity sizing: how big should the redo log be

Mon, 01 Jan 0001 00:00:00 +0000

MySQL innodb_redo_log_capacity sizing: how big should the redo log be

An undersized InnoDB redo log is a common cause of MySQL write stalls. When redo generation outpaces dirty-page flushing, checkpoint age advances until InnoDB enters synchronous flushing and user writes block.

Since MySQL 8.0.30, innodb_redo_log_capacity replaces innodb_log_file_size and innodb_log_files_in_group. Resizing no longer requires a restart, but the sizing logic is unchanged: capacity must absorb peak write rates without pushing checkpoint age into the synchronous-flush zone.

MySQL InnoDB: Database page corruption — detection and recovery

Mon, 01 Jan 0001 00:00:00 +0000

MySQL InnoDB: Database page corruption — detection and recovery

Your error log shows InnoDB: Database page corruption on disk or a failed file read. MySQL may abort the connection, mark the tablespace read-only, or crash. Page corruption is a storage-layer failure, not a query bug: a 16KB InnoDB page failed checksum validation.

The critical distinction is between secondary index corruption, which is rebuildable without data loss, and clustered index corruption, which affects the table data itself. Detect the failure, classify the damage, extract what you can, and rebuild safely.

MySQL long-running transactions: detecting and killing the silent blocker

Mon, 01 Jan 0001 00:00:00 +0000

MySQL long-running transactions: detecting and killing the silent blocker

Threads_running climbs. A DDL operation that should take seconds is stuck for minutes. Storage grows steadily with no matching data increase. The cause is often a single idle transaction in INNODB_TRX, holding row locks and an MVCC read view long after its last query finished. It does not appear in the slow query log and may have no active query string. Left alone, it blocks InnoDB purge, inflates the history list, and can trigger a metadata lock cascade that fills the connection pool.

MySQL metadata lock cascade: how one ALTER TABLE freezes a whole table

Mon, 01 Jan 0001 00:00:00 +0000

MySQL metadata lock cascade: how one ALTER TABLE freezes a whole table

You run an ALTER TABLE to add an index. Seconds later, health checks fail for queries that touch only that table. Other tables work fine. CPU and disk are idle. The connection pool fills. SHOW ENGINE INNODB STATUS shows no row lock waits. The culprit is a metadata lock cascade, not InnoDB contention.

This happens when DDL requests an exclusive metadata lock (MDL) on a table already protected by a shared MDL held by a long-running or idle transaction. The DDL waits. Subsequent DML on that table queues behind it. The queue grows until the connection pool exhausts and the application times out. Because the outage is isolated to one table and leaves no trace in InnoDB lock metrics, it is easy to misdiagnose as a network blip or a runaway query.

MySQL monitoring checklist: the signals every production instance needs

Mon, 01 Jan 0001 00:00:00 +0000

MySQL monitoring checklist: the signals every production instance needs

Most production incidents involving MySQL stem from missing observability into specific internal subsystems. Connection saturation, buffer pool thrashing, and replication lag can degrade a healthy cluster into an outage if the right signals are not monitored.

This checklist serves as a tiered operational reference for senior engineers and SREs. It defines the exact metrics needed to keep MySQL instances available, fast, and reliable, scaling from basic survival metrics to expert-level internals. Use it to validate existing dashboards or baseline monitoring for a new deployment.

MySQL monitoring maturity model: from survival to expert

Mon, 01 Jan 0001 00:00:00 +0000

MySQL monitoring maturity model: from survival to expert

Most MySQL monitoring stacks grow reactively. A team starts with a liveness check and disk alerting. An incident exposes a gap, a new signal gets added, and the cycle repeats. The result is uneven coverage: strong in areas that caused past outages, blind in areas that will cause the next one.

This article maps MySQL monitoring into four capability levels. Use it as a self-assessment, not a checklist to max out. Every level includes signals the playbook identifies as operationally important, along with the failure modes they catch. If you are missing an entire category at your current level, that is your highest-priority gap.

MySQL online DDL still blocking: ALGORITHM, LOCK, and the copy phase

Mon, 01 Jan 0001 00:00:00 +0000

MySQL online DDL still blocking: ALGORITHM, LOCK, and the copy phase

Online DDL in MySQL is not lock-free. Even with ALGORITHM=INPLACE or ALGORITHM=INSTANT, and even with LOCK=NONE, every online DDL operation passes through a brief window where it upgrades its metadata lock (MDL) to exclusive. That window is short, but it is real, and it is the single most common reason an “online” schema change still stalls a busy production table.

MySQL OOM-killed: buffer pool, per-connection buffers, and the kernel killer

Mon, 01 Jan 0001 00:00:00 +0000

MySQL OOM-killed: buffer pool, per-connection buffers, and the kernel killer

MySQL disappears. Application logs fill with connection timeouts. Uptime resets to near zero. In the kernel log: oom-kill: task mysqld and its anonymous RSS. Orchestrators mark the pod OOMKilled (exit code 137). To the application, this is a crash. To the kernel, mysqld was the largest memory consumer and the system ran out.

OOM kills are gradual, then catastrophic. Memory pressure builds as connections open, temp tables materialize, and dirty pages accumulate. The buffer pool is the obvious consumer, but the killer often enters through the back door: a connection burst multiplies per-thread buffers, a container limit sits too close to the buffer pool size, or Transparent Huge Pages block reclamation. Map all allocators to prevent recurrence.

MySQL Opened_tables climbing: table_open_cache and open-files pressure

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Opened_tables climbing: table_open_cache and open-files pressure

Steadily climbing Opened_tables with Open_tables pinned near table_open_cache means the instance is churning table file descriptors instead of reusing them. Each cache miss opens a table, consumes a file descriptor, and adds metadata lock overhead. The first symptom is usually sporadic query latency during peak traffic. Left unchecked, the instance exhausts its file descriptor limit and returns errors like Too many open files or refuses connections.

MySQL privilege change auditing: GRANT, REVOKE, and unexpected SUPER

Mon, 01 Jan 0001 00:00:00 +0000

MySQL privilege change auditing: GRANT, REVOKE, and unexpected SUPER

Unexpected privilege changes are high-impact, low-frequency events. A spike in Com_grant, an unknown account with SUPER, or a GRANT ALL outside a change window can signal privilege escalation, misconfigured automation, or migration debt. This guide covers how to detect, diagnose, and remove unauthorized grants and deprecated SUPER privileges in production.

What this means

MySQL increments status counters for GRANT, REVOKE, CREATE USER, ALTER USER, and DROP USER. The counters are cumulative; compute deltas over fixed intervals instead of reading absolute values.

MySQL purge lag from an idle transaction: the slow bleed

Mon, 01 Jan 0001 00:00:00 +0000

MySQL purge lag from an idle transaction: the slow bleed

Your OLTP workload slows over hours. No single query dominates the slow log. CPU and disk I/O are not saturated. Yet read latency climbs, throughput drifts downward, and the undo tablespace grows. The culprit is often a single idle transaction that opened a read view and never closed it. Under InnoDB’s default REPEATABLE READ isolation, that transaction pins the purge boundary. Every subsequent write appends to the undo log. The history list length grows without bound, and all MVCC reads traverse longer version chains.

MySQL query plan regression: when a query gets slow after a deploy or ANALYZE

Mon, 01 Jan 0001 00:00:00 +0000

MySQL query plan regression: when a query gets slow after a deploy or ANALYZE

A query that completed in milliseconds yesterday now times out. The deploy was clean, or maybe someone ran ANALYZE TABLE during maintenance. Slow_queries climbs, Handler_read_rnd_next jumps, and the same application code suddenly kills the database. The optimizer has changed how it executes one or more queries, and the new plan is far more expensive.

Plan regressions are sharp step-changes, not gradual growth. They correlate tightly with specific events: a schema migration, an index drop, a bulk load that triggered auto-recalc, a manual ANALYZE TABLE, or a MySQL version upgrade. The cause is usually stale or shifted statistics, an optimizer behavior change, or a missing index the optimizer can no longer use.

MySQL relay log filling the replica disk: Relay_Log_Space and recovery

Mon, 01 Jan 0001 00:00:00 +0000

MySQL relay log filling the replica disk: Relay_Log_Space and recovery

When SHOW REPLICA STATUS shows Seconds_Behind_Source climbing and Relay_Log_Space growing, the SQL thread is not applying events as fast as the IO thread fetches them. Normally MySQL purges each relay log after the SQL thread finishes it, so total disk usage stays bounded. When apply cannot keep up, files accumulate. If the partition fills, the IO thread stops. No new events arrive, lag becomes unbounded, and if the source purges binary logs before recovery, the replica requires a full resync.

MySQL Replica_IO_Running / Replica_SQL_Running not Yes: replication stopped

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Replica_IO_Running / Replica_SQL_Running not Yes: replication stopped

SHOW REPLICA STATUS\G (MySQL 8.0+) or SHOW SLAVE STATUS\G (MySQL 5.7) shows Replica_IO_Running or Replica_SQL_Running is not Yes. The IO thread may hang in Connecting, or the SQL thread may stop with an error. Replication no longer makes progress. Lag grows without bound, and if the source purges binary logs before recovery, you face a full rebuild.

Unlike many MySQL problems, a stopped replication thread does not self-heal. The SQL thread halts on the first apply error and waits for operator intervention. The IO thread retries automatically, but only up to the configured retry limit. Seconds_Behind_Source can show 0 even when the IO thread is disconnected and the SQL thread has consumed the relay log, so lag alone is not a reliable health signal.

MySQL replication lag spiral: Seconds_Behind_Source growing without bound

Mon, 01 Jan 0001 00:00:00 +0000

MySQL replication lag spiral: Seconds_Behind_Source growing without bound

You check SHOW REPLICA STATUS and see Seconds_Behind_Source is not just elevated; it is climbing. Five minutes ago it was 60. Now it is 120. The replica is not catching up; it is falling further behind.

This is a replication lag death spiral. The SQL apply thread (or threads) cannot replay events as fast as the source generates them. Relay logs accumulate. Lag compounds because longer apply queues mean larger effective transactions, longer lock holds, and greater exposure to blocking queries on the replica.

MySQL Seconds_Behind_Source is unreliable: measuring real replication lag

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Seconds_Behind_Source is unreliable: measuring real replication lag

You run SHOW REPLICA STATUS and see Seconds_Behind_Source = 0. Both replication threads say Yes. You assume the replica is current. During a source write burst, you fail over and discover hours of missing transactions. This happens because Seconds_Behind_Source is not a real-time lag measurement. It is a timestamp diff between the replica’s clock and the timestamp of the event the SQL thread is currently applying. When the SQL thread reaches the end of the relay log, the metric snaps to zero even if the I/O thread has not fetched newer events. When the SQL thread stops, it returns NULL, which most lag alerts ignore. When the source was idle and then bursts, it jumps to the idle duration even though real lag is minimal. With parallel replication, it oscillates based on worker timing. For failover-critical decisions, you need a different measurement strategy.

MySQL Select_full_join > 0: joins running without an index

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Select_full_join > 0: joins running without an index

Select_full_join is climbing. In an OLTP system, this counter should stay at zero. Any sustained nonzero rate means at least one query is executing a join without a usable index, effectively performing a full cross-product scan on the joined table. The damage scales with table size. A single unindexed join against a million-row table can stall an otherwise healthy instance.

This is not a buffer pool problem or a connection storm. It is a query correctness issue. The optimizer cannot find an index to resolve the join predicate, so it reads every row of the referenced table for each row of the driving table. In MySQL 8.0.20 and later, the execution engine uses hash join for these cases (block nested loop was removed), but the underlying pathology remains identical: missing index, catastrophic read amplification.

MySQL semi-synchronous replication stall: commits hanging on ACK

Mon, 01 Jan 0001 00:00:00 +0000

MySQL semi-synchronous replication stall: commits hanging on ACK

Commits suddenly take 10 seconds or more and then time out. On the MySQL primary, Threads_running climbs while the commit rate flatlines. A moment later, commits resume, but Rpl_semi_sync_source_no_tx is ticking upward. The primary is waiting for a semi-synchronous replica to acknowledge receipt of binlog events, and that ACK is not arriving fast enough or at all.

Confirm the stall, find the missing ACK, and decide whether to fix the replica path or temporarily fall back to asynchronous replication.

MySQL slow after restart: buffer pool warm-up and the cold cache

Mon, 01 Jan 0001 00:00:00 +0000

MySQL slow after restart: buffer pool warm-up and the cold cache

You restart MySQL for maintenance, crash recovery, or a failover. Seconds later, query latency jumps from sub-millisecond to tens or hundreds of milliseconds. Disk read I/O saturates. The buffer pool hit ratio that normally sits above 99% has collapsed. There are no runaway queries, no lock waits, and no checkpoint stalls. The server is simply cold.

After any restart, the InnoDB buffer pool starts empty. Every data page read misses memory and goes to disk, so Innodb_buffer_pool_reads spikes. If the adaptive hash index is enabled, it is empty too. The server may accept client connections long before the pool is ready to serve production traffic. The issue looks like a capacity crisis, but it is usually transient and expected. Distinguishing a cold cache from a true capacity problem keeps you from chasing the wrong fixes during an incident.

MySQL slow commits with idle CPU: redo and binlog fsync pressure

Mon, 01 Jan 0001 00:00:00 +0000

MySQL slow commits with idle CPU: redo and binlog fsync pressure

Slow commits or transaction timeouts with idle CPU usually mean the bottleneck is in the durability path, not the query engine.

With innodb_flush_log_at_trx_commit=1 and sync_binlog=1, every COMMIT triggers an InnoDB redo log fsync() and a binary log fsync() before returning to the client. These are sequential, latency-bound operations. When storage cannot drain fsync requests as fast as the server generates them, commits queue up while the CPU waits.

MySQL slow queries: from Slow_queries to the slow log to the fix

Mon, 01 Jan 0001 00:00:00 +0000

MySQL slow queries: from Slow_queries to the slow log to the fix

When Slow_queries climbs but the slow log is empty, or long_query_time is still at the default 10 seconds, you have an instrumentation gap. Slow_queries increments regardless of whether slow_query_log is ON, so a rising counter with no log detail is a dead end.

This guide moves from the status counter to slow log configuration, then to performance_schema digests, and finally to resource signals that explain why queries slowed. The goal is to help you decide in minutes whether you face a query plan regression, lock contention, buffer pool pressure, or a configuration gap.

MySQL Sort_merge_passes climbing: filesort and sort_buffer_size

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Sort_merge_passes climbing: filesort and sort_buffer_size

Sort_merge_passes climbing fast is the signature of filesort spilling to disk. You notice it in SHOW GLOBAL STATUS, along with rising disk I/O on the tmpdir partition and a growing Handler_read_rnd. Queries that returned in milliseconds now take seconds.

The absolute value of this cumulative counter is meaningless. Rate of change is what matters. A rapid climb within hours means sorts that used to stay in memory are now writing chunks to disk and merging them in multiple passes. Each pass costs disk I/O and adds latency. Active threads pile up behind slow sorts, driving Threads_running higher and the Questions rate lower.

MySQL stuck in InnoDB crash recovery: why startup hangs after an unclean shutdown

Mon, 01 Jan 0001 00:00:00 +0000

MySQL stuck in InnoDB crash recovery: why startup hangs after an unclean shutdown

After an unclean shutdown, mysqld may look healthy: the process is running, port 3306 accepts connections, and SHOW GLOBAL STATUS LIKE 'Uptime' counts up. Yet applications time out, SELECT 1 hangs, and the error log prints InnoDB recovery messages. InnoDB is replaying the redo log to make data files consistent, rolling back incomplete transactions, merging the change buffer, and running purge. Until that finishes, queries are rejected.

MySQL tablespace fragmentation: reclaiming space with OPTIMIZE TABLE

Mon, 01 Jan 0001 00:00:00 +0000

MySQL tablespace fragmentation: reclaiming space with OPTIMIZE TABLE

InnoDB tablespaces fragment after bulk deletes, updates, and purge cycles. Logical data volume shrinks, but the on-disk file retains allocated extents because InnoDB returns freed pages to its internal free lists rather than the OS. For file-per-table .ibd tables, that unused space is visible as DATA_FREE in information_schema.TABLES and can be reclaimed with OPTIMIZE TABLE.

Tables inside the shared system tablespace ibdata1 do not shrink at the OS level. Freed space returns to InnoDB’s internal free lists, but ibdata1 never releases extents back to the filesystem. If the goal is reclaiming disk space on a full volume, OPTIMIZE TABLE against tables in ibdata1 is ineffective. The rest of this guide covers identifying reclaimable tables, running the rebuild safely, and verifying the result.

MySQL Threads_connected vs Threads_running: which one to actually alert on

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Threads_connected vs Threads_running: which one to actually alert on

Threads_connected counts every client holding an open socket, including idle pooled connections. Threads_running counts only threads actively executing a statement. Paging on the first while ignoring the second is a common postmortem mistake. Threads_running relative to CPU cores is the real load signal. The gap between the two metrics diagnoses failure modes. Connection count still deserves a ticket, but rarely a page.

MySQL Threads_created climbing: thread cache churn and missing pooling

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Threads_created climbing: thread cache churn and missing pooling

When the rate of Threads_created climbs to tens or hundreds of new threads per minute, the thread cache is not absorbing your connection churn. In a pooled deployment this rate should stay near zero. Every cache miss pays the full cost of OS thread creation and initialization, plus a TLS handshake if require_secure_transport is enabled. The first visible symptom is usually connection latency spikes or CPU time diverted to thread management.

MySQL TLS and encryption monitoring: unencrypted connections and weak auth plugins

Mon, 01 Jan 0001 00:00:00 +0000

MySQL TLS and encryption monitoring: unencrypted connections and weak auth plugins

MySQL 8.0 generates self-signed certificates on startup and enables TLS by default, but this does not guarantee encrypted traffic. Clients must explicitly request TLS over TCP, and misconfigurations often leave sessions in plaintext without errors. Long-lived accounts frequently remain on mysql_native_password despite its deprecation in MySQL 8.0.34, while caching_sha2_password requires secure transport or RSA key exchange on first connection, causing sporadic authentication failures when client drivers lack encryption configuration. These risks are invisible to standard availability monitoring. This guide covers the status variables, Performance Schema tables, and account audits needed to detect unencrypted connections and weak authentication plugins in production.

MySQL undo tablespace growing: ibdata1 bloat and undo truncation

Mon, 01 Jan 0001 00:00:00 +0000

MySQL undo tablespace growing: ibdata1 bloat and undo truncation

Disk space alert: /var/lib/mysql/ibdata1 is 40 GB larger than last week, or the undo_*.ibu files in MySQL 8.0 are consuming terabytes. Table sizes from information_schema do not justify the growth. This is almost always undo log accumulation from a blocked purge thread.

Undo logs let InnoDB reconstruct older row versions for MVCC and transaction rollback. The purge thread deletes them when no active transaction needs them. If one long-running transaction holds a read view, purge stops. History list length then grows linearly with write rate, and the undo tablespace expands to hold the backlog. In MySQL 5.7, that space lands in the system tablespace ibdata1, which never shrinks. In MySQL 8.0, separate undo tablespaces can truncate automatically, but only after the history list drains and the purge thread frees rollback segments. Until then, the disk fills.

MySQL Waiting for table metadata lock: diagnosing the DDL stall

Mon, 01 Jan 0001 00:00:00 +0000

MySQL Waiting for table metadata lock: diagnosing the DDL stall

You run SHOW PROCESSLIST and see a wall of threads in Waiting for table metadata lock. An ALTER TABLE that should finish in seconds hangs for minutes. Queries against one table stop returning, the application connection pool drains, and CPU and disk look calm. This is a metadata lock (MDL) stall. It is a queueing failure, not resource exhaustion. Left alone, it cascades into connection exhaustion and a partial outage where every other table continues to work normally.

MySQL: got a packet bigger than 'max_allowed_packet' bytes - causes and fixes

Mon, 01 Jan 0001 00:00:00 +0000

MySQL: got a packet bigger than ‘max_allowed_packet’ bytes - causes and fixes

ER_NET_PACKET_TOO_LARGE (MySQL error 1153) drops the connection immediately. The payload crossed a limit on the client, the server, or a replica. A common mistake is tuning only the server: a source set to 256 MB still aborts a 20 MB payload if the mysql client is at its default 16 MB. Replicas add their own hard and soft ceilings. This guide shows how to identify which boundary failed, raise it without a restart where possible, and prevent recurrence.