How to HARDEN a Linux web server

Complete theoretical strategy to securing your web servers, from network layer down to application layer.

I’m not going to spell out every little command. I simply list the steps you have to do and let you research them for yourself. The basic thing to keep in mind is that some steps will vary depending on your Linux distribution (CentOS vs Ubuntu), what kind of stack you have (LAMP vs LEMP), and what control panel you’re using.

NOTE:

• If you’re on shared hosting or managed VPS hosting or some cloud panel, there’s a good chance you don’t have to mess with most of this. And there’s already default security applied!

STEP #1 – configure network firewall

Firewalls are the first line of security

Firewalls are to control access in and out of your server. Quite simply, they are literally the mechanism that blocks packets from going in or out of your server. You can think of them as the security guard at the front door.

• Good visitor? – ALLOW
• Bad visitor? – DENY
• Visitor wants to go to the bar? – ALLOW
• Visitor wants to go through the window? – DENY

The first line of defense is not to let them in! You have to decide which ports or services are open. Maybe web server port is open, but email ports are not. Maybe SFTP is open, but FTP is not. Maybe SSH is allowed but only to certain visitors. Maybe users from USA & Europe can access all ports, but users from China are blocked from some ports.

Depending on how you like to run your security, you can decide which firewall software to install and how you want to configure it.

• Lazy admins like a one-size-fits-all configuration that fits most scenarios.
• Relaxed admins like a safe setting that lets all services and ports work, but they manually stand by to block abusive visitors.
• Paranoid admins block as many visitors and ports as possible, only carefully allowing when necessary.

Most web servers come with firewall disabled or set very lax. This is to help provision the server much quicker in the beginning, so you don’t have to keep manually opening ports. But once the server is provisioned, you should enable the firewall immediately.

Dumb firewalls

These are basic firewall tools. Good to learn how to configure and manage. Opening and closing ports. Allowing and deny IP’s. That kind of thing. I call them “dumb” firewalls because they’re completely manual. They only do what you say. They aren’t smart enough to analyze incoming traffic and intelligently ban repeat offenders or abusive visitors that hog resources.

• By default, all Linux distributions come with iptables to manage firewall. And all other firewall tools are simple easier ways of managing iptables.
• CentOS typically comes with firewalld to manage firewall (in easier manner).
• Ubuntu typically comes with ufw to manage firewall (in easier manner).
• gufw – stands for “graphical ufw”. You can use this one instead since it has a graphical user interface but I think legit admins will never touch it.

Smart firewalls and brute force protection

I call them “smart firewalls” because they make firewall management so much easier. They usually come with helpful graphical interface, and can monitor and log traffic in ways that make more sense. Can also notify you of issues in a more intuitive way. They’re also intelligent enough to automatically ban abusive visitors. Smart firewalls make your security work so much easier because you can allow certain services and ports to stay open but don’t have to sit by the computer manually banning IP’s.

The two common mechanisms for brute-force protection are called Intrusion Detection System (IDS) and Login Failure Daemon (LFD). IDS blocks attacks based on a known signature list. LFD looks for repeated attacks and bans their IP. Some “smart firewalls” come with IDS and/or LFD built in. Other tools are only IDS or LFD. In real world use, even either of these alone can be enough if your server isn’t a massive hack target.

There is also Intrusion Prevention System (IPS) which blocks attacks before they reach your web server. But this obviously means having another proxy server in front of your existing one, and usually extra complication and increased costs.

• Fail2ban – Technically not a firewall, it does only IDS. f2b is very common, and theoretically bans repeat offenders. There are many configuration guides for it.
• Config Server & Firewall – I really like this one and it’s commonly used in CentOS servers. Friendly GUI firewall and also comes with built-in IDS/LFD.
• …and many more!

There are so many different firewall solutions out there. Each claiming better effectiveness, easier to use, or more flexibility. Just use whichever one the sys-admin is most comfortable with.

You can also pick whichever firewall is common for your server stack and has easy configuration guides. If you go with an uncommon one, you’ll need more help configuring it since there will be fewer guides to assist.

Extra reading if you’re curious:

• Firewall vs IDS vs IPS (ipwithease)
• IDS vs IPS: What’s the Difference? (DNSstuff)
• IDS vs. IPS: What is the Difference? (Varonis)
• Fail2ban and Unwanted Robots, Crawlers and Spiders (gauss development)

STEP #2 – securing SSH access

Basically, you secure the SSH (the almighty access to all server commands and files) by limiting its access. Here are some ways…

Change default SSH port from 22.

Luckily, you probably don’t have to do this as most server companies already use a non-standard SSH port. With that in mind, it’s known by hackers that their datacenter IP ranges all use the same non-default port. Should you change from that standard non-standard port? I personally don’t worry about it, but you’re welcome to if you want.

• Make up whatever port number you want and change it in /etc/ssh/sshd_config.

Create and use sudo users instead of “root” user.

All Linux servers come with a “root” user by default. It has complete and full access to anything and everything. The idea is you shouldn’t be using and giving out “root” user access on a daily basis.

If there’s people who need to make admin-level changes on the server, they should be doing it from a “sudo user”…which basically means a user that has “super user” rights…instead of always using the “root” user.

The practical use in this case is when you have many users accessing the same server and you need to give out access. It’s kinda dangerous to just pass the root password around. So it’s better if you create temporary user accounts by adding new sudo users to the server’s sudoers file. Once their work is done, you can revoke access by removing that user.

I personally don’t bother with this as me and my team are the only ones to mess in my client servers.

Use SSH keys and disable password authentication.

This is widely considered the most secure (and convenient) way to give out server access. You do it using public and private SSH keys. Anyone requesting access will have to provide a public SSH authorization key which is then added to the server and immediately removed afterwards. No root password is passed around. And any public keys accidentally exposed to the public don’t put the server at risk.

Don’t forget to disable password authentication when you use SSH keys. (This is also done from /etc/ssh/sshd_config.) Just make sure you have SSH access before you disable password access, or else you’ll be locked out of your server.

STEP #3 – changing default ports

Should you change your ports?

This is something many admins like to do because it feels to them like “hiding the front door”. They’ll change SSH port from the default 22 to something like 2220. And they’ll do the same for FTP, SFTP, SMTP, etc.

The main advantage is that automated bot scans looking for vulnerable ports will pass over your server. This will secure you from day zero attacks that haven’t been patched yet. Other advantages are that you can access your server from within restricted networks that typically block those ports.

The disadvantage is that it offers no extra security from hackers who are specifically targeting your server. And also that most automated software are preconfigured for the standard ports. You’ll commonly have to explain to other admins or contractors what your non-standard ports are.

STEP #4 – preventing web server abuse

Resource abuse is the most common effect of server attacks.

If you don’t already know, most hacks today are more about resource-abuse rather than security breach. In other words…most hacks today are designed around overwhelming your server with requests rather than actually breaking into your server and accessing files.

They do it by bombarding your services, making fake requests, leaving connections open, etc. It’s the equivalent of angry citizens bombarding their politician’s phone-lines. It causes service interruption more so than breaching access.

Using safe PHP limits.

Your web server should limit its connections to prevent abuse. You can find these settings in your php.ini for each PHP version you have installed.

• Max connections – 1k is a good amount. Don’t worry about setting it higher. If you have more visitors, the server closes the older connections. This setting may also coincide with max SSL connections.
• Max connections per IP – 10 is a safe bet but you might need more for certain services.
• Max execution time – the leaner your site, the lower you need. Most connections only need a few seconds, but we typically allow up to 60-300 seconds to allow for slower processes like imports and uploads.
• Max input vars – usually 1k is safe, but some applications may need 3k-10k.
• Memory limit – I would stick to 128mb if you can. A lean WordPress site uses only half of that. If you have more plugins and things, then you’ll need more.

The lower your limits, the less likely you are to have resource hogs. But if you set too low, then some services may not work correctly. The best practice is always to have super clean code. So you can set these limits really low. I also like setting lower limits so I’m notified sooner (via broken site) when memory is being hogged.

Update to latest PHP versions, get rid of old PHP versions.

Updating your PHP version gives you better security and also performance. Typically only the most recent PHP versions are maintained. At the point of this writing, it’s PHP 7.2, 7.3, and 7.4. Any older PHP is not maintained and not secured against recent attacks.

STEP #5 – securing frontend applications

Most server attacks are due to application vulnerabilities.

Most servers are hacked through the software running on them and never because of the server security itself. This is because the vulnerabilities are in your application, not necessarily your server.

Think about it like this…doors are usually main security points of any building, right? It’s because they allow traffic in and out. (You never see guards protecting a closed wall.) It’s the same way for servers. Usually, everything is completely closed up except for the applications. Because DUH…applications have to let people in and out. It lets people request info from your server and also give information to your server (like in forms, passwords, text input, etc).

And within those requests for information or submitting information is where applications can become vulnerable. The server can’t completely secure that since it doesn’t know what applications to specifically secure for.

Securing WordPress applications

Now there are endless possible web applications and I can’t account for every single one. I’ll only cover the WordPress ones here, and the common vulnerabilities specific to WordPress.

• XML-RPC protocol – if you’re not using it. Block it.
• wp-login.php – you should protect this against brute force attacks.
• Update regularly – keep your WordPress core, themes, and plugins updated. Update everything you have installed, not only the ones you use!
• Audit your themes/plugins – don’t install anything that was coded like crap, or hacked/nulled plugins that you got from some “cheap plugin site”. Don’t use anything coded by low-level programmers.
• Install security plugins – security plugins aren’t necessary if your server is secured well but they can help with specific functions and alert you if there’s a potential issue. I like Wordfence for occasional malware scanning and login notifications, but you can use something else if you like.
• Be cautious of any forms you have on your site. Any place that information can be inputted into your site can leave you vulnerable to an injection attack.
• Prevent PHP execution in your uploads directory.

STEP #6 – common sense practices

These things are everyday practices that promote good security.

• Keep server, packages and modules updated.
• Use strong passwords – do I have to explain this?
• Don’t use the same password for everything – don’t use same password for database, WordPress admin, server root, email, Facebook, bank account, PayPal. Or else if they get one, they will get into all the others!
• Remove unused modules – anything you don’t use, disable or remove.
• Keep applications updated – this goes for the server as well as your frontend applications.
• Have backups – it’s often easier to restore a site than clean up a hack. Even if you do want to clean up, having a backup makes it easier to find irregularities.
• Have a developer and sys-admin available – security really isn’t the place to DIY. You should experts available to audit your setup before issues happen, and then also to clean up if you do get hacked.

STEP #7 – configure web application firewall (WAF)

Web Application Firewalls (WAF) are different from Network Firewalls.

“What?! Another kind of firewall?!” Yeah yeah, I know. It’s confusing.

• Network firewalls – are standard firewalls that control access between ports and IP’s. They open and close ports, allow and deny IP’s.
• Web Application Firewalls – are software-level firewalls that can limit traffic within open ports.

Dammit, I suck at explaining. Let’s take port 80 (HTTP) and 443 (HTTPS). You can’t block those ports because they’re needed for web browsers to function. So on a network level, all traffic is allowed through….regardless if it’s good or bad. Now the question is…how do we stop the bad traffic from getting through the open port?

In this case, we rely on a software-level “firewall” (which technically isn’t a firewall from the traditional networking definition)…it’s an application that checks all incoming traffic and stops it if it’s bad. How does it know if it’s bad? Many little things, like user agent and bot signature, excessive number of requests and such. WAF’s are essential for blocking application-level DDOS attacks. Since they block by the agent and type of attack rather than blocking by port and IP.

If I could make an analogy. Network Firewall is like deciding which doors in your building to allow entry. Web Application Firewall is deciding which of the entry doors to put a bouncer. And as we already know, putting that bouncer there will slow down all incoming traffic because they each get checked (regardless of good or bad).

I usually avoid WAF if I can help it.

Anyway…the key thing to know is I hate using WAF as much as possible because it really hurts server performance. Sure, some of them out there say they’re fast (and I believe it) but you’ll always be faster without one.

But that doesn’t mean you should copy what I do.

• Low-maintenance server – better to put totally rock-solid security and adjust only if clients have problems with blocked API’s and such.
• High-maintenance server – use relaxed settings for better performance. If you’re often touching the server (like I am), you can react just fine.
• DNS-level WAF – there are security services out there that do it at the proxy. For example, you can use Cloudflare’s security rules. Very powerful stuff and won’t increase server load.

Learn how to read your server logs:

• How to analyze and interpret Apache Webserver Log (LinuxConfig)

For those of you using ModSecurity (which can be slow):

• ModSecurity Performance Recommendations (Trustwave)
• Ways to improve performance of your server in ModSecurity 2.5 (Packt)

If you want to do it manually, you can block bad bots with htaccess or with Cloudflare firewall rules.

Learn more about:

• Web Application Firewall vs Network Firewall (ipwithease)
• A Guide to Web Application Firewall vs. Network-Level Firewall (TrustRadius)
• Blocking bots: Why WAFs fall short and how to fix it (DataDome)
• Is Your WAF Up to the Bot Management Task? (DynamicDNS)
• Bot Manager vs. WAF: Why You Actually Need Both (radware)
• Bad Bots Mitigation (Sucuri)
• Application Layer DDoS Attack (Cloudflare)

If you guys are really curious, I could probably write a whole guide dedicated to DDOS attacks and firewall security.

That’s it?

Yeah, that’s it. It’s not that much work to set up.

Especially when it’s like your 200th server. Most “server hardening” guides out there are just only basic steps to configure firewall and secure SSH, nothing else. It’s best if you can go a few steps beyond that as most security issues nowadays are a matter of resource-abuse, rather than security breach.

The problem is when you get hacked or new hacks come out, you’ll manually have to adjust things. Maybe you have to release some restrictions to let traffic in, or put extra restrictions to block certain kinds of abusers.

• 3rd-party API? – maybe you use some 3rd-party service that makes many requests to your server. You’ll have to allow them through to avoid service interruption.
• Day zero hack – maybe there’s a new hack that harasses your server on a certain layer, port, or frontend software attack. You’ll have to manually block it.

Over time, after you’ve managed so many servers…you might forget which servers have which restrictions applied. Sure, you can read notes but it’s faster just to check. Also…it can take time to read through logs and find the issue. Especially when you have so many sites and so much traffic sifting through one server.

Security is a process, not a destination.

True server security is having a fast response time to attacks.

It’s not about having a server so secure that nothing ever gets through. Automated software makes it easier but you still need trained human personnel who know where to look when problems arise, and how to efficiently stop those attacks when they happen.