Recovering from a HACKED web server (Linux)

Comprehensive guide on how to recover from server attacks (whether inbound or outbound). NOTE: this guide is written for WordPress users but can be applied to any CMS.

I wrote this security guide to be as helpful as possible. It should help you detect and repair at least 99% of the hacks out there. I repair around 20 servers every year due to hackers, intrusions, and other interruptions caused by attacks. This cheatsheet was originally compiled for my personal use but has since been re-written to be digestible for even newb server admins. (In the midst of complicated server jargon guides out there, I figured to help the community by writing something more actionable for real-world use.)

The only requirement of this guide is that you know how to get to the command line. Also, I don’t cover all nuances of server security here or list every command for every linux distro. I’m more often using CentOS (RHEL) rather than Ubuntu (Debian). It’s up to you to look up alternative commands if that’s what you need.

First thing to do when your server gets hacked…

OH NO! YOU JUST GOT HACKED!! WHAT DO YOU DO?!! WHAT DO YOU DO?!

Usually, you find out your server got hacked because your datacenter or provider has network-restricted you. Or maybe you find out because of failed services, or defaced websites showing malware. It’s a scary situation as you don’t know what’s wrong and the immediate reaction is to panic. What’s most annoying is that it always seems to happen at the worst time (e.g. big project, vacation, wedding, medical illness).

1. Hire an expert

Seriously, don’t mess around. If you don’t know what you’re doing or have critical sites/client stuff, just hire an expert and be done with it. Now is not the time to play DIY. I advise you not to continue venturing on your own unless you really like being an IT paramedic and being responsible for other people’s lives.

2. Figure out what’s being hacked

• Is it an inbound attack?
• Or is it an outbound attack?
• Is it just malware and defaced website or altered data?
• How much access did the hackers get into your server? (Just a backdoor/script running from from user directory? Or is it a root-level intrusion?)
• Did your webhost or datacenter limit your network connections?
• If the damage is really bad, do you have a plan to temporarily restore critical client sites?

3. Be prepared to build a new server

In many cases, if your server is hacked that badly, it’s faster to rebuild a new server from scratch than to waste time trying to find all the hacks and repair all the damaged services and config files in the server. It’s also not wise as you’re not 100% sure whether it’s still compromised somewhere.

4. Restore from backup or no?

Many people get lazy and try to resolve hacks simply by restoring an older backup. Sure, this can work if the data hasn’t changed much. Just restore the backup and then quickly harden your server/sites to block the impending attack. But it’s not an option if you don’t have backups or the backup data has changed since. Quite often, we don’t notice an attack until much later from when the server was breached. So if you notice it too late, the clean backups may have already been overwritten.

Recovering from INBOUND ATTACKS

This is when outside machines/servers are attacking YOUR server. Some of these attacks actually try to gain entry into your server but others only intend to disrupt services by overwhelming your server. Regardless of their intentions (to get in or not), all of these attacks eat up server resources rendering it unable to load your website for real website visitors.

Different kinds of inbound attacks:

• Brute force – multiple attempts in rapid succession at guessing your admin passwords and gaining entry through your login pages or other connection protocols (like XML-RPC for WordPress).
• Flood attacks – denial of service (DOS), the even stronger distributed denial of service (DDOS), or SYN FLOOD attacks. These attacks specifically target the server on different ports and protocols, requesting many open connections (beyond the server’s limit). It’s the equivalent of mass-calling someone’s phone-line to make it unavailable for legitimate callers.

These attacks can be randomly targeted or specifically-targeted. The random ones are annoying and erratic. The specifically-targeted are most impactful (they don’t stop until your site goes down). It helps to not have controversial material, exposed IP’s on the internet, or making enemies on the internet. Sometimes, you just can’t help it. It might just be because you have an ecommerce site and those make attractive targets.

Detecting inbound attacks:

• Server running slow – this is an obvious sign you might getting hacked. Especially if you haven’t changed anything else on the site and traffic is still the same.
• Check for high server (CPU) load grep processor /proc/cpuinfo | wc -l. Unnecessary if your webhosting control panel already has a GUI for this. Anything at or above the number of CPU cores you have is considered really high (i.e. load of “5” when you only have 4 cores). High CPU usually means an attack at network level (bombarding services).
• Check for high memory usage cat /proc/meminfo or top. High swap messages in your control panel at random intervals are also an obvious indicator. Sometimes the attacks will erratic and you’ll just have to scan logs. High memory usually means an attack at software level (bombarding php scripts).
• Check connections per IP netstat -ntu|awk '{print $5}'|cut -d: -f1 -s|sort|uniq -c|sort -nk1 -r. Up to 50 connections from one IP can be normal, anything over 100 is suspicious. If you see many single connections but coming from the same subnet, check 2nd half of this guide.
• Alternate commands to check connections per IP tail -n 10000 yourweblog.log|cut -f 1 -d ' '|sort|uniq -c|sort -nr|more and netstat -n|grep :80|cut -c 45-|cut -f 1 -d ':'|sort|uniq -c|sort -nr|more. Use these if the previous ones didn’t help.
• Check for syn connections netstat -n | grep :80 | grep SYN
• Check logs for failed login attempts cat /var/log/secure (RHEL, Centos, Fedora) or cat /var/log/auth.log (Ubuntu, Debian).
• Check for WordPress wp-login brute-force attack (current day) grep -s $(date +"%d/%b/%Y:") /usr/local/apache/domlogs/* | grep wp-login.php | awk {'print $1,$6,$7'} | sort | uniq -c | sort -n
• Check for WordPress XMLRPC attack (current day) grep -s $(date +"%d/%b/%Y:") /usr/local/apache/domlogs/* | grep xmlrpc | awk {'print $1,$6,$7'} | sort | uniq -c | sort -n
• Then ban the most offending IP’s and/or disable the ports and services that are being attacked.

Stopping inbound attacks:

• Ban IP’s – you can block the most offending IP’s through your firewall or security plugin. It might also be helpful to install security software like fail2ban that automatically scans your logs and bans the most obvious IP’s. Sure, there are some debates about its efficacy since it may slow down your server and also unable to detect all kinds of attacks.
• Disable ports/services (OPTIONAL) – this is another good tactic to remove attacks by taking away their attack point. Just make sure it isn’t a port or service that you’re actually using.
• Enable security (firewall) plugins – the point of using security plugins is to block attacks more efficiently than your current firewall solution. Again, the real risk isn’t that these attacks actually get in but that they use up server resources.
• Enable security (firewall) services – another way of blocking attacks is to rely on a security service. Usually they operate at the DNS level, and you send all your traffic through their proxy servers so that they can police incoming requests. This is especially important when you have really complicated attacks that low-level plugins cannot handle. For example, layer 7 DDOS attacks are able to send tons of requests to your computer from many different machines. Because it’s a botnet and not just one computer, you cannot ban a single IP. You’d need a much more sophisticated security service (that probably relies on multiple computers) with the capacity to quickly process these requests and let legitimate traffic through without affecting their page load [too much].

I wish I could be more clear about how to ban IP’s and disable ports/services but it really depends on what firewall you have have. The most common ones I’ve come across are:

• iptables (command-line)
• ConfigServer
• ModSecurity
• Firewalld
• Lua-Resty-WAF

See which one you have and if you DON’T have one, well it’s time to install one then! Then look up its documentation to see: which ports/services are open, where the block/ban lists are, how to make changes. Don’t forget to restart it after you make changes. (Note to myself to update this distinguish between network vs application firewalls.)

Reference links:

• Linux Logs Explained – Plesk
• How to Check if your Linux server is Under DDoS Attack? (more info on checking subnets) – Hivelocity
• Server under DDOS attack – How to find out IPs? – serverfault
• 12 Critical Linux Log Files You Must be Monitoring – euroVPS
• iftop – A Real Time Linux Network Bandwidth Monitoring Tool – TecMint
• Is there any logs in Linux which tells if some port has been denied – serverfault
• Basic DoS/DDoS Mitigation with the CSF Firewall – LiquidWeb
• Checking the Domain Access Logs for Abuse and Resource Usage – KnownHost

Recovering from OUTBOUND ATTACKS

This is when there’s a hack or malware script on your server, using your server to hack other servers. Typically, they take over your server, using it and its resources as part of their botnet to target other servers. Pretty malicious and evil, I know! In doing this, they can clever deploy tons of servers to brute force or DDOS for them instead of having to pay for all those servers themselves. It’s very diabolical! And worst of all, aside from using up your server resources, it gets YOUR SERVER and YOUR IP reputation trouble. It’s the equivalent of a criminal using a stolen car to rob banks.

Unlike other hacks, you cannot fix these on your own time. Quite often, I’ll have clients that have a hacked site but aren’t in a rush to fix it since it’s not as important as the others. Well, guess what? These hacks typically cause other server owners and datacenters to complain to your datacenter that you’re hacking them. At this point, your hardware vendor has no choice but to do the responsible thing by limiting YOUR network connections. That means restricting some of your ports and services to limit the damage, which will affect all sites on the server. So sure…it might not be an important site that was hacked but it will affect all other sites. A definite quandary if you’ve got other client sites on there. So what do you do? Fix the outbound attack ASAP and so your datacenter can lift the network restrictions.

Different kinds of outbound attacks:

• Same as the inbound attacks but now it’s your server that’s doing the hacking. Brute force, flood attacks, email spam, etc.

But this time, we don’t diagnose it the same way. We don’t look to see how many connections we have with each IP because the idea isn’t to ban other IP’s. The strategy now is to see which illegal processes are running, find out their location, kill them (from running), and delete them. And then most ideal is to find the hole where they got in and fix or remove that vulnerability as well.

Detecting outbound attacks:

• Check outbound connections netstat -nputw and see which ones look suspicious. If you see too many lines, try netstat -nputw | less (PS: you can exit less command with q). If you see any “stealth” processes on the right side, take note of their process ID number (aka “PID”).
• Can also check for stealth processes directly ps -ef | grep stealth.
• Find location of stealth process lsof -p 12345 | grep cwd and lsof -p 12345 but replace 12345 with actual process ID. Now you know which directory to clean. You should also check /tmp directory as stealth processes often run files from there find /tmp | grep -i stealth.

Stopping outbound attacks:

• Clean the directory manually (using your eyes to detect bad files), or plugins (like Wordfence) to scan the site.
• Check recently modified files within last 24 hours find /directorypath -mtime -1 -ls using path to home directory of hacked user account or wherever the hack was. (Of course, change the path or timeframe as needed). More info on adjusting this command here. Then go in there and delete or fix files! (NOTE: if the list of files is too long, you can use the pipe command to output them to a file.)
• Change passwords – if you saw backdoor scripts and Adminer during your cleaning process, it’s probably a good idea to change all control panel, admin, and database passwords. If you also used any of these passwords for your email, PayPal, eBay, Facebook accounts…I recommend changing them as well. (Hackers often cross-check your passwords against other online services.)
• Kill the process, using kill if you know the process ID (kill 12345) or pkill if you know the process name (pkill processname). This will kill the PID, as well as anybody running the process under that ID.
• Find and close the vulnerability – time to figure out how they got in. Almost always, it’s a vulnerable theme or plugin (probably one that has a form in it). Usually you have to check the modified files. But what if you didn’t detect the hack until many days later? Unfortunately, it’s really hard to know as you probably won’t have the time to read all the logs or maybe the logs showing how they got in have already been deleted. All you can do from here is update all your 3rd-party extensions and keep a close eye on the site. If it gets hacked again, immediately check the recently-modified logs.

Reference links:

• Kill Linux Processes Easier with pkill – How-to Geek
• Linux Stealth Process – serverfault
• How To Identify The Bad Processes On A Hacked Linux Box – ma.ttias.be
• How can I monitor all outgoing requests/connections from my machine? – UNIX & LINUX
• How do I get the path of a process in Unix / Linux – stackoverflow
• Detecting WordPress Outbound Bruteforce attack – nixtree

Recovering from INJECTION, MALWARE, DEFACEMENT ATTACKS

Malware or defacement attacks are usually the most obvious and most low-level ones. You can see it when your website all of the sudden starts redirecting to another site (probably with ads and questionable material), or maybe your site itself is showing ads, weird pop-ups and triggering security warnings. The worst is when they actually change the data in your database, changing your content and even payment gateway API (re-routing incoming payments to their financial accounts instead of yours!).

Different kinds of malware/defacement attacks:

• Website redirects to another one
• Website has ads or weird messages/content that you didn’t put
• Website content altered and now has links in it that you didn’t put
• Website redirects incoming customer payments to the hacker’s financial account instead of yours
• Website stops working or is broken
• Login page hacked into a “phishing page” and sends info from user login attempts to the hacker

In case you’re wondering of how they got there in the first place…WELL, it usually has to do with some vulnerable plugin or theme that you had running on your site. And to be more specific, it’s almost always some plugin that has a form somewhere or allows users to input information. Quite often, these forms don’t properly “sanitize” the data (sanitization means to disallow illegitimate data)…which then allows the hacker to plug vulnerable code into the database (aka “SQL injection”). These SQL injections are used to run commands that can return information like user names and passwords, payment gateway API’s, or output hack code to a php file (backdoor scripts). And of course, these backdoor scripts are usually placed inpublicly-allowed directories like image and upload directories which allow php execution. Why do plugins sometimes fail to sanitize data? It’s either one or a mix of both lazy and incompetent coding.

The thing that many people don’t understand about security: hackers got in because YOU installed or activated some vulnerable plugin that then let them in. It’s not because your security system failed.

Detecting malware/defacement attacks:

Usually pretty obvious as you can see your website functions have gone awry but now is the time to check the usual places. The first 3 steps must because checked and corrected first.

• Check htaccess – open it up and see if there are new lines in there that you didn’t add.
• Check wp-config (or other CMS config) – open it up to see if the site URL was changed in there.
• Check database – if using WordPress, go to the wp_options table and look at the “site address” and “WordPress address” rows. If it’s got the wrong URL in there, change it back. If they changed all the urls in your database (uncommon), you’ll have to manually change all those strings back.
• Check theme files – go into your active theme directory and look around for weird files. Also check the functions.php file to see if any malicious functions were put in there.
• Check plugin files – same thing as above but in your plugin directories. This option is often not realistic if you have too many directories to look through.
• Check uploads directory (or other public directories) – many hacks and scripts will hide (and execute) from these directories because these are open to the public. Would be smart to block php execution from the uploads directory.
• Check plugin settings in WordPress – log into your WordPress admin (or other CMS admin) and check all settings to see if they’ve been changed. What you’re looking for is any place where they might have changed sensitive info, like putting their PayPal email instead of yours, putting their logo or site URL instead of yours, having your backups go to their remote storage instead of yours, etc…the possibilities are endless. Be thorough and check everything over carefully!
• Checking for base 64 encrypted code – hackers use this to hide their code. So you can’t read and see the exact strings to search for. No worry at all…we learn how to search base64 code below…

Use find and grep to search for these strings. (But beware that there are legitimate uses for base64.):

• base64_decode
• gzinflate(base64_decode
• eval(gzinflate(base64_decode
• eval(base64_decode

Stopping malware/defacement attacks:

• Run a malware scanner – run something like WordFence as it has the best malware database and also notifies you of which files are changed and which files don’t belong there. Honestly, this should have been the very first step as soon as you can get into your site admin. But I mention all the other since you have to be at least pass over it with your eyes and know how to do some of this stuff manually.
• Check recently modified files – using commands I already shared above.
• File comparisons – this is so time-consuming and probably not absolutely necessary unless this is a truly critical site and you need to make sure no other hack or vulnerability is on the server. Simply compare side-by-side between current site backup and one from a date you know is clean. If you’re clever, you’ll know how to do this quicker with code editors.

Reference links:

• SQL (Structured query language) Injection – imperva
• Creating Backdoors using SQL Injection – INFOSEC
• Use SQL Injection to Run OS Commands & Get a Shell – NULL BYTE
• Hiding Linux processes for fun + profit – Sysdig
• The Trickster Hackers – Backdoor Obfuscation and Evasion Techniques – imperva
• https://hackforums.net/index.php – resource for hackers (you can see what they scheme up)

Recovering from ACCESS BREACH

This is when hackers have gained access into your websites (or web server) and its content. Even scarier is when they have their own admin accounts or gained access to yours, even worse—have root-level access! Server accounts, FTP, database, email, website admin, and so forth. It’s scary stuff!

Different kinds of access breaches:

• Website admin – they have admin rights and can change info as they please.
• FTP access – they’re able to upload files and scripts to your server, or also download stuff off of it.
• Database access – they’re able to download or change data, even inject code into files.
• Email access – able to read, or send emails from your server.
• Server access – with access to the server, they can do any or all of the above and even take complete control of your server. Once a hacker has root access and enough time, they could theoretically create so much damage and chaos that it’d be much faster for you to rebuild a new server from scratch than to try repairing their damage. The risk is that even if you found 99.99% of their hacks but still left one backdoor open, they could let themselves back in again.

Detecting access breaches:

• Check who is logged in (and where from) w, the hacker might be logged in and working as you speak. Take note of the usernames used and their IP location. Most likely if they’ve gotten this far, they’ve gained root access. Do NOT try to kick them out just yet! You don’t know how much access they have and trying to kick them out now might cause immediate retaliation (further hindering your recovery process).
• Check login history last. Useful to see who has previously logged in. Again, take note of usernames and IP location. You should be very suspicious if the login history is empty (that means somebody is hiding their tracks!)
• Check command history history, shows you all the recent commands used (also stored at ~/.bash_history). Look for wget or curl commands used to install malicious software/packages. Again, if you see nothing that means somebody is hiding their tracks.
• Check for high use processes top, look at the top cpu-use processes. Hackers with root level access typically use as much server resources as possible to hack other servers, send email spam, or mine for cryptocurrency. If you don’t recognize a process, try lsof -p 12345 or strace -p 12345 (replacing “12345” with the actual process ID number). Lsof will show all the files run by a process (super useful).
• Check all processes ps, ps aux, ps auxf. Each one shows more info than the last. I personally prefer “ps aux”. Here, you can see all running processes and can take note of any that you don’t recognize. TIP: the more often you run this command the better you’ll get at spotting strange processes.
• Check network usage iftop shows processes sending/receiving data, along with their source and destination. Any processes abusing your network with DOS or spam will show at the top.
• Check listening connections lsof -i or netstat -plunt, look for any “LISTEN” or “ESTABLISHED” processes that you don’t recognize. It’s good to check for listening processes as they don’t consume much CPU to get noticed in “top” but are used by hackers to send commands to the server. Again: use the “lsof -p” command to look up processes if you don’t know what they do.

Stopping access breaches:

• Shut off server, and hire someone – if this is way above your level. Shut off the server and don’t turn it back on until you have an experienced admin there to quickly remove and reseal so the hacker can’t re-access.
• Disable SSH from all IP’s except your own – do this after you turn server back on. Make sure you’re the only one logged in.
• Block all ports and services – start limiting your firewall and re-allow things one by one only when you’re sure they’ve been verified.
• Kill and remove processes – like you did with previous steps.
• Change passwords – to everything.
• Search for new admin accounts – hackers often create new admin accounts for themselves once they get in. Or they might do clever things like increasing another user’s rights to full admin, or disguising an admin account as an official “support” account for your webhost/software. Look carefully and remove or adjust all accounts. They often create multiple admin accounts if they go through the trouble at all.
• Copy everything to new server – I’m sorry but just about everyone will tell you it’s irresponsible to continue working off a previously compromised server. There’s no telling how much damage was done and not smart to risk it. You’re safer off copying everything to a fresh install. With that said, this option is quite drastic and may not be absolutely necessary if your site was only vulnerable at the user-level and some backdoor scripts. The sentiment is more for when people had their server breached at the root/admin level.

Reference links:

• How To Tell If Your Linux Server Has Been Compromised – Bash Prompt
• How to List Users in Linux – Linuxize
• Tutorial: How check if your linux server has been hack. – spiceworks
• How do I deal with a compromised server? – serverfault
• Outgoing brute force attack from my IP – WebHostingTALK

Quick thoughts on server security

I know most of you will be asking this question here but I really don’t want to write a post-in-a-post, so I’ll leave you with a few quick thoughts.

• Firewall is best handled at the server level or even DNS level (by 3rd party security service). Using application-level plugins would be resource heavy, slow down legitimate visitors, and not as comprehensive.
• Malware scanning is theoretically best (most resource-efficient) done at the server level BUT…the problem is that applications are so complicated within their own plugins and extensions that they need their own application-specific malware scanners to be thorough enough for zero-day attacks.
• The best server-level malware scanner is probably ImunifyAV (free) or Imunify360 (paid). With that said, I don’t use it. So that goes to show how much you really need it. I’m sure it’s great for catching little server oddities and email spam and what not. You can use them if you’d like an easy GUI to do your malware scanning and perhaps auto-scheduling options.
• The best application-level malware scanner (for WordPress) is Wordfence. It’s got the best signature database, most thorough and protective. Wordfence catches more hacks, malware, and intrusions than any other WordPress security/malware plugin IMO. It would be so awesome if Wordfence could design a server-level plugin to use on WordPress-oriented servers.
• Maldet is a total waste of time. I can’t tell you the number of times this thing has failed to find the exact hack causing all the problems. At best, it finds some old hacks in your email files and that’s it. It’s terrible for zero-day attacks or real-world use IMO.
• When it comes to firewall plugins, you need SMART [ADAPTIVE] firewall plugins. With a “dumb” [manual] firewall, it doesn’t adapt to attacks and no settings are applied unless you manually apply them yourself. Smart firewalls can read logs and ban IP’s on the fly based on their behavior.
• Most security plugins (both server and application) out there are junk. Just plugins with fancy marketing and logos that barely do even as much as the free ones.
• The best way to handle website/server security AND prevent attacks and security mechanisms from slowing down your sites…is to use best practices and other typical checklists security tasks. I’ll do a post on that later. Honestly, you should be learning from real sys-admins showing you their favorite security configs. I don’t even consider myself a senior admin…so if you’re trying to learn from me, you’re already doing it wrong!
• Here’s another cool way to deal with security vulnerabilities…by scanning for vulnerabilities during the development process. (Cool service: RIPSTECH) The only issue is that it’s probably very expensive and the only developers who would care to use this are the ones being responsible enough with their coding that they probably won’t need it.