Sample incident: MariaDB OOM kill on a shared cPanel node

Early one Saturday morning, monitoring on cpanel-host paged the on-call engineer. The symptom was the kind every cPanel operator has seen: every WordPress site on the box was returning a database error, and systemctl status mariadb reported the service had exited.

What alerted

The first signal was a 500 burst on the public sites. The second was the MariaDB unit going inactive. The kernel had killed it:

journalctl -u mariadb --since "10 min ago" | tail -20

mariadb.service: Main process exited, code=killed, status=9/KILL
mariadb.service: Failed with result 'signal'.

status=9/KILL plus a missing mysqld process is the OOM-killer signature. The next check confirms it:

dmesg -T | grep -i 'killed process'

[Sat May 10 03:14:02 2026] Out of memory: Killed process 18432 (mysqld)
total-vm:6182144kB, anon-rss:5421008kB, file-rss:0kB, shmem-rss:0kB

Five gigabytes of resident memory on a node where the buffer pool was configured for two. Something had grown.

What we ran

The first job was to get the sites back up. systemctl start mariadb brought the daemon back; the buffer pool warmed up in under a minute. The second job was to find which database had grown.

mysqlcheck --all-databases --check 2>&1 | tail

wpdb_146.wp_options                              OK
wpdb_146.wp_postmeta                             OK
wpdb_022.wpgt_options                            OK
wpdb_1104.wprt_postmeta                          OK

Tables were healthy. The growth was elsewhere: the wp_options autoloaded set on wpdb_146. A plugin had been writing transient sessions without expiry for weeks. The autoload payload was 380 MB, loaded into memory on every request.

The fix

Two changes landed the same morning. First, prune the runaway autoloaded options on wpdb_146:

DELETE FROM wp_options
WHERE autoload = 'yes'
  AND option_name LIKE '%_session_%'
  AND option_name NOT IN ('siteurl', 'home');

Second, raise the InnoDB buffer pool with headroom for the working set and a hard ceiling on tmp_table_size:

[mysqld]
innodb_buffer_pool_size = 4G
tmp_table_size          = 64M
max_heap_table_size     = 64M

systemctl restart mariadb and the working set settled at 3.1 GB resident. No further OOM kills.

What we changed in the use case

ServerGuard's use case for "MySQL OOM on cPanel node" is implemented today: it detects status=9/KILL in the unit log, restarts the daemon, identifies the largest autoloaded option set across all WordPress databases on the node, and surfaces a prune SQL for human approval.

It does not yet edit my.cnf automatically. Buffer-pool changes are gated behind explicit operator sign-off. That is upcoming.

Background noise

This incident was not adversarial. But the same morning, the auth.log on the box showed the usual grind:

sshd[12044]: Failed password for root from 203.0.113.114 port 51220 ssh2
sshd[12047]: Failed password for root from 203.0.113.92  port 51221 ssh2

A pair of attacker subnets that have been hammering SSH on every node we run for months. They are unrelated to the OOM. We no longer publish the addresses verbatim — naming them adds nothing the next operator cannot get from their own logs.