Locked out of cPanel SSH: VNC, iptables, and the way back in

The terminal hangs. You hit Enter again. Nothing. You try a different SSH client. Nothing. You try from your phone's hotspot, on a different ISP, with a different public IP, and SSH connects on the first try. That is the moment you know the server is fine and you have locked yourself out. Somewhere between the last config change and the next breath, your admin IP became persona non grata to CSF, or to lfd, or to Imunify360, or to a plain iptables rule that you wrote three months ago and forgot.

This is the recovery use case we run when it happens. It assumes nothing about which firewall layer is blocking you. It walks through verifying that the lockout is real, getting back in via the provider's out-of-band console, flushing the firewall safely, investigating before rebuilding, and re-enabling protection without re-locking yourself in the process. It also covers the boring, unsexy work you do before the next lockout: the five-minute audit that turns a two-hour midnight scramble into a one-line csf -a and a restart.

We will use cpanel-host for the affected server and 198.51.100.10 as the admin IP throughout. The commands are exactly what we run on cPanel/WHM hosts running AlmaLinux 8 or 9 with CSF and optionally Imunify360.

First: is it really a lockout?

The most common false alarm is "SSH is broken" when SSH is fine and the network path is broken. Before you reach for the recovery console, spend ninety seconds confirming the failure mode. The recovery path for a firewall lockout is dangerous; the recovery path for a routing problem at your ISP is "wait fifteen minutes". Do not run the wrong one.

Three quick checks from another machine on a different network:

# 1. DNS resolves to the expected IP.
dig +short cpanel-host
 
# 2. ICMP reaches the host.
ping -c 4 cpanel-host
 
# 3. The TCP handshake on the SSH port.
nc -vz cpanel-host 22

Three states tell you what is happening:

• Connection refused on port 22 means sshd is not listening (the daemon crashed, or you moved it to a different port and forgot). This is a server-side problem, not a firewall lockout, and the recovery is "start sshd" not "flush iptables".
• No response, packets dropped silently means a firewall is eating your SYN packets. This is the classic CSF/lfd/iptables lockout. The recovery flow in this post is for this case.
• Connection reset by peer mid-handshake is usually a TCP wrappers / hosts.deny issue or sshd's own AllowUsers rejecting you. Different recovery; still requires console access if you cannot SSH in to fix it.

If you have a second machine on a different ISP you can SSH from, tcpdump confirms the diagnosis instantly:

# On a third host (not the locked-out admin, not the server):
tcpdump -ni any "host cpanel-host and port 22"
 
# Then from the locked-out admin machine, try SSH.
# If you see SYN packets arrive and never get SYN-ACK back,
# the firewall on cpanel-host is dropping them.

ICMP also being blocked confirms a full firewall lockout rather than a per-port rule. If ping works but SSH does not, only the SSH path is blocked, probably an lfd temporary block, not a manual iptables -A INPUT -j DROP.

Recovery via your provider's out-of-band console

Every serious VPS or cloud provider gives you a way into the host that does not depend on SSH. You log into the provider's web console, click a button, and a browser-based terminal opens connected to the serial port or VNC framebuffer of the running server. The firewall on the box has no idea this connection exists, because the connection does not go over the public network. It goes through the hypervisor's management plane.

The provider you are on dictates the exact path. The common ones we have used:

DigitalOcean

Open the Droplet, go to Access > Recovery Console. A new browser tab opens with an HTML5 VNC session attached to the Droplet's console. Log in as root with the password you set at Droplet creation (or via passwd root later). If you only ever used SSH keys and never set a root password, this is the moment you regret it. This is why item one on the pre-lockout audit at the end of this post is "set and store a root password".

DigitalOcean's recovery console is keyboard-only, slow to refresh, and pastes badly. Type carefully. Once you have shell, jump to the "recovery commands" section below.

Vultr

Server detail page > View Console. Same browser-based VNC, slightly nicer pasting. Same login flow: root + password.

Linode (Akamai)

Linode calls their console Lish. From the Cloud Manager, open the Linode and click Launch LISH Console. Lish is a web-based serial terminal, not VNC, so it pastes well and runs over SSH on the back end (you can also SSH to lish-<dc>.linode.com directly with your Linode account credentials, which is faster than the browser). Glish is the graphical equivalent if your boot needs a real framebuffer. For our purposes, Lish is enough.

AWS Lightsail and EC2

Lightsail: instance detail > Connect using SSH opens a browser SSH session via AWS-managed keys; this works only if sshd is up. If you are fully locked at the firewall, you need EC2 Instance Connect (if enabled on the AMI) or the Serial Console (for Nitro instances). On older EC2 instances without Instance Connect or Serial Console, the unpleasant fallback is to stop the instance, detach the root EBS volume, attach it to a recovery instance, mount it, edit the firewall config files directly, then reattach and boot. It works; it is not fast.

Hetzner

Hetzner Cloud: server detail > Console opens an HTML5 console. For dedicated servers, the Robot panel has a remote KVM option that takes a few minutes to provision.

When your provider does not have a console

Some budget bare-metal providers do not ship out-of-band access by default. The fallback options, roughly in order of preference:

• IPMI / iDRAC / iLO: physical-server management interface, usually on a separate IP. If your provider gave you credentials, this is the equivalent of plugging a monitor and keyboard into the rack.
• KVM-over-IP on demand: some providers will roll a portable KVM cart to your server on request, for a fee, within a service window.
• Provider support ticket: ask them to flush the firewall for you. They will usually do this with proof of account ownership, but the queue can be hours, and you should not depend on it.
• Reboot into rescue mode: most providers offer a rescue-image boot that mounts your root filesystem at /mnt and gives you a shell from a known-good environment. From there, edit /etc/csf/csf.allow, remove problematic rules in /etc/sysconfig/iptables (or /etc/csf/csf.deny), then reboot back into your normal system.

The painful lesson, learned the hard way more than once: verify your out-of-band access works before you need it. The five minutes of clicking through the recovery console on a quiet Tuesday afternoon is the cheapest insurance you will ever buy.

The recovery commands once you have a console

You are at a console prompt, logged in as root, looking at a shell on the locked-out server. The order of operations matters. Flushing the firewall first and asking questions later is correct, because every minute you spend diagnosing from the VNC console is a minute your other admins cannot SSH in either. Get back to a known-safe state, then investigate.

Step 1: stop the firewall safely

# Disable CSF temporarily. csf -x leaves the binary in place
# but flushes its rules and prevents lfd from re-adding them.
csf -x
 
# Belt and braces: flush iptables directly.
iptables -F           # flush all chains
iptables -X           # delete any custom chains CSF created
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
 
# If the host is dual-stack, do the same for ip6tables.
ip6tables -F
ip6tables -X
ip6tables -P INPUT ACCEPT
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT

This leaves the server with no firewall at all. That is fine for the next five minutes. cPanel servers do not need a firewall to stay alive long enough for you to fix the configuration; they need a firewall to stay alive long enough to survive the public internet for the next month. Five minutes of exposure to finish recovery is an acceptable trade. If the server is actively under attack right now, jump to Step 4 and whitelist your admin IP before you go any further, then come back and complete Step 2 and 3 from a real SSH session.

Step 2: verify SSH is reachable

From your laptop, try SSH again:

ssh -v admin@cpanel-host

If you connect, stop and resist the urge to immediately rebuild the firewall. The firewall is what locked you out; rebuilding it blindly will lock you out again. Drop into the server, leave the VNC console open as a safety net, and investigate from a proper terminal where you can paste and scroll.

If you still cannot connect with the firewall fully flushed, the problem is not the firewall. Check systemctl status sshd, ss -tlnp | grep ssh, and the /var/log/secure for clues. sshd may not be running, or it may be listening only on a different port, or TCP wrappers / AllowUsers may be filtering you. Those are different problems; the rest of this post is about firewall lockouts specifically.

Step 3: investigate before rebuilding

You have shell, you are not locked out, the firewall is flat. Now figure out which layer actually banned you. Three log files tell the story between them:

# Current CSF temporary blocks (empty after csf -x, but the file
# /etc/csf/csf.deny may show permanent ones):
cat /etc/csf/csf.deny | grep -v "^#"
 
# Recent lfd activity: rate-limit triggers, brute-force blocks,
# distributed attack detections.
tail -n 200 /var/log/lfd.log
 
# System log for Imunify360, fail2ban, manual iptables changes,
# or kernel-level netfilter drops.
tail -n 200 /var/log/messages
 
# Imunify360 own log if it is installed.
tail -n 200 /var/log/imunify360/console.log

Look for your own admin IP 198.51.100.10 in any of these. The three common patterns:

• You banned yourself: a stray csf -d 198.51.100.10 from a typo, or csf.deny edited by hand, or a CSF rule change that did not include your IP in csf.allow. The fix is the next step.
• lfd banned you: usually LF_SSHD triggered by repeated failed logins (often from a stale SSH key being retried by an automation agent), or LF_DISTATTACK flagging your IP as a coordinated source. The lfd log will say exactly which trigger fired.
• Imunify360 banned you: signature match on a request from your IP, or an OSSEC-style behavioural rule. The Imunify360 console log will name the rule.

Less common but real:

• A cron job that runs csf -df on a schedule and the whitespace-handling bug in older CSF versions silently strips your allowlist. Pin CSF to a current version.
• A second admin on the team added a deny rule without checking csf.allow first.

Name the root cause before you proceed. "I do not know" is a valid answer for now, but write it down. You will want to come back and finish the analysis after the dust settles.

Step 4: whitelist your admin IP first

Before re-enabling CSF, make sure your IP is permanently allowed:

# Add to csf.allow with a comment.
csf -a 198.51.100.10 "Permanent admin - on-call laptop"
 
# Verify it is in the file.
grep "198.51.100.10" /etc/csf/csf.allow
 
# If you have a second admin IP, add it too. Single admin IPs
# are a single point of failure.
csf -a 203.0.113.7 "Permanent admin - bastion VPS"

A line in csf.allow survives both csf -df (deny flush) and csf -tf (temp flush). It does not survive someone editing csf.allow by hand and removing it, which is exactly how some of these lockouts started in the first place. Treat csf.allow like a production config file: change it through CSF commands, log every change, and review it at least quarterly.

Step 5: restore CSF carefully

# Re-enable CSF with the new allowlist in place.
csf -e
 
# Watch lfd in real time for the next few minutes. If your IP
# triggers any rule, you will see it immediately and can ctrl-C.
tail -f /var/log/lfd.log

Open a second SSH session from your admin machine right now, while the VNC console is still open. If the second SSH session works, the allowlist is correct. If it does not, your IP is being blocked somewhere else: Imunify360, fail2ban, or a hand-written iptables rule that survived iptables -F because it is loaded from a unit file on every boot.

Common second-layer culprits to check when CSF looks correct but you are still being blocked:

# Imunify360 allowlist
imunify360-agent ip-list local add 198.51.100.10 --comment "admin"
 
# fail2ban, if installed
fail2ban-client unban 198.51.100.10
 
# Persistent iptables rules outside CSF
cat /etc/sysconfig/iptables 2>/dev/null
systemctl list-units --type=service | grep -E "iptables|netfilter"

Only once a fresh SSH session from your admin IP succeeds and stays up for a few minutes should you close the VNC console.

Move SSH off port 22 (and away from default alternates)

Most lockouts we trace back to admin error happen on hosts running sshd on port 22. The reason is straightforward: port 22 attracts a constant background of scanner traffic, lfd's rate-limit triggers fire dozens of times a day, and once you are tuning lfd thresholds in production you eventually tune them wrong.

Moving sshd to a non-default high port is not security by obscurity in any meaningful sense (a determined attacker scans all 65,535 ports) but it is noise reduction. Less log noise means lfd can be configured with tighter thresholds without false positives, and tighter thresholds make the real brute force attempts visible instead of buried.

The pattern we use:

# /etc/ssh/sshd_config: listen on both ports during transition
Port 22
Port 15600
 
# Test that 15600 works before removing 22
systemctl reload sshd
 
# From your admin machine
ssh -p 15600 admin@cpanel-host

# /etc/csf/csf.conf: add 15600 to TCP_IN
TCP_IN = "20,21,25,53,80,110,143,443,465,587,993,995,15600"

# Reload CSF after editing csf.conf
csf -r
 
# If Imunify360 is installed, allow the port there too
imunify360-agent config update '{"FIREWALL": {"port_blocking_mode": "ALLOW"}}'

Once port 15600 works end-to-end, remove Port 22 from sshd_config, remove 22 from TCP_IN in csf.conf, reload sshd and CSF, and verify your csf.allow still contains your admin IPs.

Why we picked 15600 and not 2222 or 22222: scanners hit the common-alternate ports too. Botnets that scan 1-65535 will find you regardless, but the dumb scanners that try a hardcoded list of "alternate SSH" ports will not. The gain is small but real. Port 22 sees thousands of attempts per day on a public IP. Port 15600 on the same IP usually sees zero.

Document the new port in your team wiki. The number of times a new admin has tried to SSH into a server, hit silence on port 22, assumed the server was down, and woken someone up at 02:00 is embarrassing.

Out-of-band recovery checklist

This is the audit we run on every new server before it goes into production. Twelve items, takes five minutes per host, and turns every future lockout from a panic into a footnote.

• Provider VNC / console works. Actually log into it once and type a command. Do not assume.
• Root password is set, stored in the team password manager, and rotated when staff change.
• A second admin IP is in csf.allow. Different ISP from the first.
• csf.allow contents are documented somewhere off-server (password manager note, internal wiki).
• sshd listens on a non-default port and the port is in TCP_IN in csf.conf.
• You have tested SSH from at least two different admin locations in the last 30 days.
• LF_TRIGGER_PERM in csf.conf is set sanely (0 means temporary blocks always; 1 and up means permanent blocks after N triggers. Pick deliberately).
• RESTRICT_SYSLOG and RESTRICT_UI reviewed.
• Imunify360 (if installed) has the admin IPs in its allowlist too.
• No iptables-services running alongside CSF. CSF assumes it owns iptables; two managers in the same kitchen end badly.
• Cron jobs that touch firewall config are reviewed quarterly.
• A documented recovery runbook exists. Print this post if you like.

Why we always whitelist a second admin IP

A single admin IP is a single point of failure. Your home internet goes down, your laptop reboots and grabs a new DHCP lease on a guest network, your phone hotspot is on a CGNAT range that lfd flagged, and suddenly the server you maintain is unreachable from the only IP it trusts.

The pattern we recommend on every host:

• Home IP of the primary on-call engineer
• Office IP of the team (if there is a fixed-IP office)
• Bastion VPS on a different provider. A $5-a-month VPS whose only job is to be a stable IP you can SSH from. We use one in a different region from production, owned by the agency account with 2FA on the provider portal. The bastion has its own hardened sshd config, key-only auth, and a CSF allowlist of one: the on-call engineer's residential IP plus the office.

Three whitelisted IPs across three networks means three independent failure paths must coincide before you are locked out and forced to the VNC console. We have seen one or two fail simultaneously many times. Three is rare enough that we have not had to use the VNC console for an admin-error lockout in over a year.

The five-minute pre-lockout audit

Right now, on every server you operate, run this short script. It will not modify anything; it just prints what you need to verify.

echo "=== csf.allow ==="
grep -vE "^#|^$" /etc/csf/csf.allow
 
echo "=== sshd Port directives ==="
grep -E "^Port" /etc/ssh/sshd_config
 
echo "=== csf.conf TCP_IN ==="
grep "^TCP_IN" /etc/csf/csf.conf
 
echo "=== Recent lfd blocks of admin IPs ==="
grep -E "198.51.100.10|203.0.113.7" /var/log/lfd.log | tail -20
 
echo "=== sshd is running and listening ==="
ss -tlnp | grep ssh

Replace the admin IPs with yours. If any of the five sections looks wrong, fix it now while you can still SSH in. The cost of fixing it proactively is sixty seconds; the cost of fixing it from a VNC console at 02:00 is forty-five minutes plus the cortisol.

If you also operate cPanel servers behind CSF and lfd, the CSF, lfd, and Imunify360 conflict post walks through the layering rules we use to stop these three tools from fighting each other. And if your servers are also dealing with the constant background of SSH brute-force traffic that drives lfd to over-block in the first place, our writeup of an 8,127-attempt SSH brute-force night covers the rate-limit configuration we settled on.

How ServerGuard handles this

ServerGuard's relationship to admin lockouts is a useful test of how we talk about capability honestly, because the scenario splits cleanly into two halves: prevention, where SGuard is genuinely strong today, and mid-incident recovery, where we are not yet autonomous and we will not pretend otherwise.

Prevention: what we do today. SGuard subscribes to CSF and Imunify360 events on every monitored server. When a block of any kind is proposed (by lfd, by Imunify360, by a manual csf -d, by a Claude-suggested remediation) SGuard checks the target IP against the server's allowlist of admin and bastion IPs. If the target IP is on the allowlist, SGuard refuses to execute the block and writes the refusal to the audit log. This is not a feature; it is a guardrail. Admin IPs are sacred. Even if our own diagnosis model returns a recommendation to block an allowlisted IP (for example because the IP is showing brute-force-like patterns due to a stale credential being retried) the guardrail wins and the block does not execute. The on-call human gets a notification and decides.

SGuard also runs the pre-lockout audit above on a schedule for every monitored server: it checks that admin IPs are in csf.allow, that sshd is listening on the documented port, and that at least two admin IPs are configured. Drift on any of these surfaces as a warning before the lockout happens.

Mid-incident recovery: the honest limit. Once you are locked out, SGuard cannot rescue you. Our control plane reaches your server over the same SSH connection that is now blocked. The same firewall rule that locked you out locks us out. Calling the provider's VNC API to inject commands without an active SSH session is upcoming, but it is not in the product today, and we would rather say so than ship a marketing claim we cannot back. The flow in this post (VNC, csf -x, iptables -F, restore) is the recovery path with or without SGuard.

The shape of upcoming will narrow that gap. Out-of-band recovery via provider APIs (DigitalOcean, Linode, Hetzner first, then AWS), an escape-hatch port on a kill-by-uid lockfile pattern, and automatic restoration of the admin allowlist after a third-party tool removes it are all coming, in roughly that order. Until they ship, the audit and the bastion VPS do most of the work, and they do it well.

If you operate cPanel servers and you want the guardrail in place before your next firewall change, join the waitlist. We are onboarding agencies one at a time and the configuration walk includes setting up the admin-IP allowlist on every server before SGuard takes any action at all.