Compromised Linux server: what to do in the first hour

A CPU pinned at 100% by a process nobody recognises, an abuse report from the hosting provider, a GuardDuty alert, or a web page that suddenly redirects to a casino: the Linux server has been compromised, or probably has. You got the call and have to act before anyone has a full picture. Here is the runbook I use.

You will get a minute-by-minute plan for the first hour, the commands to run at each step, the places attackers hide on Linux, and the decision you need to make at the end: clean or rebuild. Incident response, compromised-server recovery and malware cleanup are part of my day-to-day work, and under pressure the order of the steps matters more than any single tool.

Two goals that pull in different directions

You want the service back. You also want to know what happened, so it does not happen again next week. Those goals conflict, because almost every restore action (rebooting, killing processes, deleting files, restoring a backup over the disk) destroys evidence.

The answer is to do them in order: preserve first, then restore. Preservation takes about thirty minutes if you have practised it. After that, you can restore service on a new, clean server while the old one sits isolated for analysis.

The exception: if the attacker is actively destroying data, for example ransomware encrypting files right now, stop the damage first. Cut the network, and if encryption continues, stop the instance. Lost evidence is recoverable. Lost data often is not.

Rules for the whole hour

• Do not power off or reboot. Memory holds the running malware, decrypted configs, network connections and often the attacker’s commands. A reboot erases all of it. RFC 3227 puts memory and the process table near the top of its order of volatility for this reason.
• Do not trust the box. RFC 3227’s advice is blunt: don’t trust the programs on the system. ps, ls and netstat can be replaced or hooked. Use static binaries from your own media where you can, and treat everything the host tells you as a lead, not a fact.
• Write evidence somewhere else. Never save output to the compromised disk. Attach a fresh volume, or stream output over SSH to another host.
• Keep notes with UTC timestamps. What you ran, when, and why. RFC 3227 says to be prepared to testify, perhaps years later, about every action you took.
• One person types. Others watch, take notes and handle communication.

The hour at a glance

Minutes	Action	Why now
0-5	Declare the incident, start notes, pick a lead	Stops ad hoc fixes that destroy evidence
5-15	Isolate at the network edge, protect from termination	Cuts attacker access and stops spread
15-20	Snapshot every disk	Preserves the disk before you touch it
20-35	Capture memory and volatile state	Lost on reboot or process kill
35-45	Hunt persistence, miners and web shells	Finds how they stay and what they run
45-55	Rotate credentials	Assumes everything on the box is known to the attacker
55-60	Decide rebuild or clean; notify people	Sets up recovery and meets legal duties

Minutes 0-5: declare and write things down

Say it out loud in your team channel: “This is an incident.” Name one incident lead. Open a notes document and write the first entry:

2026-11-11T09:14Z  Incident declared. Host web01 (i-0abc...), 10.0.1.23.
Trigger: CPU 100% since ~08:40Z, unknown process running from /tmp. Lead: <name>.

Then ask three questions and write down the answers: What does this server hold (customer data, payment data, credentials to other systems)? What else trusts it (other servers, databases, CI)? Who has to be told if data was taken? You will need those answers at minute 55.

Minutes 5-15: isolate without powering off

The goal is to cut the attacker off and stop the host talking to anything else, while it keeps running.

On AWS

The AWS Security Incident Response Guide sets the order: capture instance metadata, enable termination protection, isolate with a security group, detach from Auto Scaling, deregister from load balancers, snapshot the volumes, and tag the instance.

ID=i-0abc1234def567890
aws ec2 describe-instances --instance-ids "$ID" > "$ID-metadata.json"

# 1. Nobody (including Auto Scaling) can terminate it by accident
aws ec2 modify-instance-attribute --instance-id "$ID" \
  --attribute disableApiTermination --value true

# 2. Take it out of service so it stops being replaced or receiving traffic
aws autoscaling detach-instances --instance-ids "$ID" \
  --auto-scaling-group-name web-asg --should-decrement-desired-capacity
aws elbv2 deregister-targets --target-group-arn "$TG_ARN" --targets Id="$ID"

# 3. Isolation security group: no inbound except your responder IP, no outbound
SG=$(aws ec2 create-security-group --group-name ir-isolation-web01 \
  --description "IR isolation" --vpc-id "$VPC" --query GroupId --output text)
aws ec2 revoke-security-group-egress --group-id "$SG" \
  --ip-permissions '[{"IpProtocol":"-1","IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]'
aws ec2 authorize-security-group-ingress --group-id "$SG" \
  --protocol tcp --port 22 --cidr 203.0.113.10/32
aws ec2 modify-instance-attribute --instance-id "$ID" --groups "$SG"

# 4. Label it so nobody "fixes" it
aws ec2 create-tags --resources "$ID" \
  --tags Key=ir-status,Value=quarantined Key=ir-ticket,Value=INC-2026-1111

If the VPC has IPv6, also revoke the ::/0 egress rule. Detaching from the Auto Scaling group matters more than it looks. AWS documents that termination protection does not stop Auto Scaling from terminating an instance that is unhealthy or being scaled in, and an isolated instance will fail its health checks. Detach it, or the group may replace it and take your evidence with it.

A security group change does not cut live connections

This catches experienced engineers. AWS documents that when you change a security group rule, existing tracked connections are not interrupted; packets keep flowing until those connections time out. An attacker’s reverse shell or SSH session can survive your isolation.

Network ACLs are stateless, and adding a deny rule breaks existing connections immediately. A NACL applies to the whole subnet, so don’t block everything; add deny rules for the specific remote IPs you see in the attacker’s connections (you will list them in a moment with ss):

aws ec2 create-network-acl-entry --network-acl-id "$NACL" --ingress \
  --rule-number 10 --protocol -1 --cidr-block 198.51.100.7/32 --rule-action deny
aws ec2 create-network-acl-entry --network-acl-id "$NACL" --egress \
  --rule-number 10 --protocol -1 --cidr-block 198.51.100.7/32 --rule-action deny

Off AWS

The same idea applies everywhere: block at a layer the attacker cannot touch from inside the host. That means the provider’s firewall, the hypervisor’s virtual switch, or the physical switch port. A firewall rule you add with iptables on the compromised host can be removed by an attacker with root.

The instance’s cloud credentials

If the instance has an IAM role, assume the attacker read its temporary credentials from the instance metadata service. In the IAM console, open the role and use Revoke active sessions. AWS then attaches an AWSRevokeOlderSessions policy that denies every session issued before that moment. GuardDuty raises UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS when those credentials are used from outside AWS, so check for it.

Minutes 15-20: snapshot every disk

Snapshot before you run anything that walks the disk, since even find can update directory access times.

aws ec2 create-snapshots \
  --instance-specification InstanceId="$ID" \
  --description "INC-2026-1111 web01 pre-triage $(date -u +%FT%TZ)" \
  --copy-tags-from-source volume

create-snapshots captures all attached volumes at the same point in time. Record the snapshot IDs in your notes. Later, analysts will attach volumes made from these snapshots to a clean forensic instance, read-only, and do the slow work there.

Minutes 20-35: collect volatile data

Attach a new, empty volume to the instance for evidence (for example 50 GB gp3), format it and mount it at /mnt/ir. Everything below goes there.

Memory first

Memory is the most volatile thing you have. Microsoft’s AVML is a static userland binary for x86_64: no kernel module to build and no compilation on the target. Copy it in from your own trusted source, not from the internet on the compromised host:

sudo /mnt/ir/tools/avml acquire --compress /mnt/ir/web01-mem.lime.compressed
sha256sum /mnt/ir/web01-mem.lime.compressed > /mnt/ir/web01-mem.sha256

AVML’s documentation notes that it cannot acquire memory when kernel lockdown is enabled. On Graviton (arm64) instances, use a tool built for that architecture, such as LiME compiled for the exact kernel on a separate machine. If you cannot capture memory, record why and move on.

Processes, connections, users

Run this as one block so the snapshot of state is as consistent as possible. Each line writes to a file on the evidence volume.

OUT=/mnt/ir/$(hostname)-$(date -u +%Y%m%dT%H%M%SZ)
sudo mkdir -p "$OUT" && cd "$OUT" || exit 1
date -u                      > 00-start-utc.txt
uptime                       > uptime.txt
ps auxwwf                    > ps.txt
ss -tupan                    > ss.txt
{ ip addr; ip route; ip neigh; } > ip.txt
w                            > w.txt
who -a                       > who.txt
last -Faiw                   > last.txt 2>&1
lastb -Faiw                  > lastb.txt 2>&1
ls -l /proc/[0-9]*/exe 2>/dev/null | grep '(deleted)' > deleted-exe.txt
cat /etc/ld.so.preload       > ld-so-preload.txt 2>&1
systemctl list-units --type=service --state=running --no-pager > services-running.txt
systemctl list-timers --all --no-pager > timers.txt
sha256sum -- * > SHA256SUMS

What to look for:

• ss.txt: established connections to IPs you don’t recognise, and listeners on odd ports. Note the remote IPs for your NACL rules.
• ps.txt: processes with random or kernel-like names ([kworker]-style names that are not children of kthreadd), anything running from /tmp, /dev/shm or /var/tmp, and shells owned by www-data.
• deleted-exe.txt: a process whose binary has been deleted from disk is a classic sign of malware cleaning up after itself.
• last.txt and lastb.txt: logins from unusual IPs or at odd hours, and the brute-force that may have preceded them.

One suspicious process, in detail

For each process that looks wrong, save its binary and context before you kill it:

PID=4242
sudo cp /proc/$PID/exe   "$OUT/pid-$PID.bin"
sudo tr '\0' ' ' < /proc/$PID/cmdline > "$OUT/pid-$PID-cmdline.txt"
sudo tr '\0' '\n' < /proc/$PID/environ > "$OUT/pid-$PID-environ.txt"
sudo ls -l /proc/$PID/cwd /proc/$PID/fd > "$OUT/pid-$PID-fds.txt" 2>&1
sha256sum "$OUT/pid-$PID.bin"

Copying from /proc/PID/exe works even when the file was deleted from disk. Search the hash in your threat intel tools from a separate machine.

Minutes 35-45: hunt persistence

Attackers who get in once want to get back in after you reboot or change passwords. These are the places to check on any Linux box, with the MITRE ATT&CK technique each one maps to.

Where	Technique	Check
~/.ssh/authorized_keys for every user, including root	T1098.004	New or unknown keys, recent mtime
User crontabs, /etc/crontab, /etc/cron.*	T1053.003	Download-and-run lines, base64 blobs
systemd units and timers in /etc/systemd/system, ~/.config/systemd/user	T1543.002	Units created recently, odd ExecStart paths
/etc/ld.so.preload	T1574.006	Should normally be absent or empty
.bashrc, .profile, /etc/profile.d/	T1546.004	Commands that run at every login
/etc/passwd	Create Account	Extra UID 0 accounts, new users with shells
Web roots	T1505.003	PHP or JSP files in upload directories

The commands, again writing to the evidence volume:

awk -F: '$3 == 0' /etc/passwd                                    > uid0.txt
getent passwd | awk -F: '$7 !~ /(nologin|false)$/'               > login-shells.txt
for d in /root /home/*; do for f in "$d"/.ssh/authorized_keys*; do
  [ -f "$f" ] && { echo "== $f"; ls -l "$f"; cat "$f"; }; done; done > authorized-keys.txt
for u in $(cut -d: -f1 /etc/passwd); do
  crontab -l -u "$u" 2>/dev/null | sed "s/^/$u: /"; done          > crontabs.txt
ls -la /etc/cron.* /etc/crontab /var/spool/cron/crontabs         > cron-files.txt 2>&1
grep -rEn '(curl|wget)[^|]*\|[[:space:]]*(ba)?sh' /etc/crontab /etc/cron.* /var/spool/cron > cron-download-run.txt 2>/dev/null
find /etc/systemd/system /usr/lib/systemd/system /home/*/.config/systemd /root/.config/systemd \
  -type f -newermt '30 days ago' -ls 2>/dev/null                 > systemd-recent.txt
find / -xdev \( -path /proc -o -path /sys -o -path /run -o -path /mnt/ir \) -prune -o \
  -type f -newermt '7 days ago' -printf '%T+ %u %m %p\n' 2>/dev/null | sort -r > recent-files.txt
find / -xdev -type f -perm -4000 -printf '%T+ %u %p\n' 2>/dev/null > suid.txt
lsattr -R /etc /usr/bin /usr/sbin 2>/dev/null | grep -E '^[^ ]*i' > immutable.txt
dpkg --verify > dpkg-verify.txt 2>&1          # RHEL family: rpm -Va

dpkg --verify lists packaged files whose checksum no longer matches the package database; a 5 in the third column of a binary under /usr/bin is worth a close look. An attacker with root can tamper with that database too, so confirm anything important by comparing against a clean image of the same release.

The immutable.txt check exists because attackers set the immutable flag so that even root’s rm fails. MITRE records the TeamTNT group modifying binary permissions with chattr.

Crypto miners and web shells

Both are quick to check, so check them on every incident.

Crypto miners (MITRE T1496, Resource Hijacking) are loud: sustained high CPU and outbound connections to mining pools. Expect a reinstaller too. MITRE records the Rocke mining group installing a cron job that downloaded and ran files from its servers, and Kinsing using crontab to fetch and run scripts every minute. Rocke also modified /etc/ld.so.preload to hide its mining software from process lists, which is why the earlier ld.so.preload check matters and why top can lie.

top -b -n 1 -o %CPU | head -20
grep -al 'stratum' /proc/[0-9]*/cmdline 2>/dev/null     # stratum is the mining pool protocol
ss -tnp state established

On AWS, GuardDuty raises CryptoCurrency:EC2/BitcoinTool.B and CryptoCurrency:EC2/BitcoinTool.B!DNS when an instance talks to known mining infrastructure.

Web shells (T1505.003) are scripts dropped into a web root that run commands sent over HTTP. Look for script files where only images should be, recently modified scripts, and the common eval and exec patterns:

find /var/www -type f -path '*uploads*' -name '*.php*' -print
find /var/www -type f -name '*.php' -newermt '14 days ago' -printf '%T+ %u %p\n' | sort -r
grep -rlE --include='*.php' '(eval|assert)[[:space:]]*\((base64_decode|gzinflate|str_rot13|\$_(POST|GET|REQUEST|COOKIE))|(system|shell_exec|passthru|exec|popen|proc_open)[[:space:]]*\([[:space:]]*\$_(GET|POST|REQUEST|COOKIE)' /var/www
grep '"POST ' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

A POST request to a file inside an upload directory is rarely legitimate. The access log line for the first such request often shows you the attacker’s IP and the time they got in, which tells you how far back to look everywhere else.

Minutes 45-55: rotate credentials

Assume the attacker has read every file on the box. Rotate, from a clean machine, everything the server could see:

• SSH keys: any private key on the server, and remove the server’s keys from authorized_keys on every host that trusted it.
• Application secrets: database passwords, API keys, SMTP credentials, payment provider keys, anything in .env files, wp-config.php or config management.
• Cloud credentials: the instance role sessions (done above), and any access keys found in ~/.aws/credentials or environment variables. Review CloudTrail for what those keys did; event history keeps 90 days of management events in each Region at no charge.
• TLS private keys if they were readable: reissue the certificate and revoke the old one.
• Human passwords for any account that logged in to the box, and session tokens for admin panels hosted on it.

Do the rotation after isolation. Rotating while the attacker is still connected just shows them the new values.

Minutes 55-60: rebuild or clean, and who to tell

Rebuild is the default

My rule is simple. If the attacker had root, or you cannot say with evidence how they got in and what they touched, rebuild. Launch a new server from a known-good image, apply your hardening baseline (my Linux server hardening checklist is the one I use), deploy the application from source control, restore data from a backup taken before the first sign of compromise, and fix the entry point before you open it to traffic. If the site is public, put the new server behind Cloudflare with the origin locked to Cloudflare’s IP ranges; the settings are in my Cloudflare guide by plan.

Cleaning in place is only reasonable when the intrusion was contained to an unprivileged account, you found the entry point, and you can account for every change the attacker made. Even then, keep the old disk snapshots. Rootkits that hook the loader or kernel can hide from every check in this post, and you will not know what you missed.

Notifying people

Work out who must hear about this, and when, with your legal adviser:

• Internally: the system owner, management, and whoever owns customer communication.
• Data protection regulators: under GDPR Article 33, a controller must notify the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of a personal data breach, unless the breach is unlikely to result in a risk to people. Other jurisdictions have their own rules and clocks.
• Customers and partners: contracts often contain their own notification terms.
• Your cyber insurer, if you have one. Read the policy’s notification and approved-vendor terms before you engage outside help.

Legal and forensic caveats

This is a technical runbook, not legal advice. A few points to settle with counsel early:

• Chain of custody. If there is any chance of legal action or law enforcement involvement, record who handled each piece of evidence and when, and keep the SHA-256 hashes you generated. Work on copies, never on the original snapshots.
• Consider bringing in a forensic firm before you touch the host when the incident involves regulated data, large losses or likely prosecution. Their findings carry more weight than yours.
• Privacy law applies to you too. Evidence will contain user data. Limit who sees it and how long you keep it.
• Don’t hack back. Logging in to the attacker’s servers or pools is likely illegal, however satisfying it would be.

NIST SP 800-61 Rev. 3, published in April 2025, frames incident response around the CSF 2.0 functions (govern, identify, protect, detect, respond, recover). It is a good basis for the written plan you should have before the next incident.

First-hour checklist

Print this and keep it with your on-call notes:

• Incident declared, one lead, notes in UTC, nobody “fixing” things ad hoc
• Server left running; not rebooted, not powered off
• Termination protection on; removed from Auto Scaling and load balancers
• Isolation security group applied; NACL deny rules for attacker IPs to kill live sessions
• Instance role sessions revoked
• All disks snapshotted, IDs recorded
• Memory captured with AVML to a separate evidence volume, hashed
• Processes, connections, logins, deleted binaries and preload file collected and hashed
• Persistence checked: keys, cron, systemd, preload, shell profiles, UID 0 accounts, web roots
• Miner and web shell checks run; suspicious binaries copied before any kill
• Every secret the server could read rotated from a clean machine
• Rebuild decision made; entry point identified or rebuild is mandatory
• Legal, regulator, customer and insurer notification needs assessed

If you need hands on keyboard for an incident like this, 24/7 incident response is part of what I do through NoDowntime Technologies.