Securing your Rails environment

Unlike what you may think, even your 'managed' account from a reputable hosting company (RackSpace, LiquidWeb, EV1, etc.) has more holes in it than Swiss cheese. Security is something you need to worry about, not your hosting provider. While your host will help you get the basic security services running, it's not their core service nor do they have an incentive to provide the full security package - you will loose the most if your server is compromised, and hence you should be the one worrying about it. I've learned this the hard way - I had several of my servers compromised in the past, and let me tell ya, it's not something you want to have to deal with. In 95% of the cases these situations can be avoided by securing your environment prior to deploying your app and that is what we will try to do here. The other 5% is unfortunately out of your control (new exploits, update patches - sometimes you just can't patch your servers fast enough before someone gets in. One of my production servers once went down within 10 minutes of a new SSH exploit being released to the public.) So let's get to it...

1. Pick your poison, err... Operating System

First off, choose an operating system you're comfortable with. I've used RedHat distributions for all of my servers (7.3, RHEL3, RHEL 4, CentOS flavors) in the past and over the years I've learned many gotchas, config file locations, log locations, etc. all of which helps me navigate and troubleshoot the systems much more efficiently. If you have a choice, stick with something you know, save your precious time for development instead! Next, pick a control panel (or lack of thereof), there are dozens of software packages which will allow you to control much of the functionality of the server via a web-interface (CPanel, Plesk, Webmin...), the choice to use one of these is entirely up to you. If you are comfortable with the command line on your server, you probably don't need it. However, if you're a newcomer to managing your own servers, a control panel like CPanel can be a great addon because you get a lot of built in functionality AND yet another level of support from the CPanel team, which is never a bad thing. In my case, my VPS came with CPanel preinstalled (they didn't have a 'just OS' option). Now, assuming your server is running, let's really get to it...

2. Disable all the fluff

Before I even begin 'securing' my server I always start by disabling everything but the core services required by the kernel. If you don't know what you're trying to protect when you build your security fence, chances are, you're not going to build it right. Furthermore, if you're using a VPS account you're usually looking at limited resources and we don't want to use them used up on irrelevant services. Hence, your first task should be to figure out exactly what you will need running on your server. In my case, my VPS came with CPanel, which by default includes a number of services I had no need for. Furthermore, I decided to offload the mail services to Google and DNS to my provider (both services are free - good deal for a bootstrap!), leaving me with the barebones of Apache / Mongrel / SQL. CPanel users can follow these steps:

  • Shutdown Exim - From experience, Exim is a huge resource hog, in 8 out of 10 cases I can trace a server load spike to Exim. To free up the server resources I decided to use GMail for my domain instead.

  • Shutdown MailQueueRunner / LogWrapper / Clamd / IMAP / POP Wrapper - Those are the reasons Exim is such a hog - all of the above services are for mail support. That's a lot of resources, kill them all!

  • Shutdown Named (DNS), Interchange, Melange, FTPD (use SFTP), etc. - If you don't use it, kill it - that's one less security hole you need to worry about (yes, every open port is a security hole, I can count all open ports on my system - can you?).

3. Create a new user account for SSH

It's bad practice to use 'root' to SSH into your server, instead you should use another username to login and then 'su -' to get root privileges. To add a new user do: useradd -g wheel admin which will add an 'admin' user and put in the 'wheel' group - allowing admin to obtain superuser rights. To set the password do passwd admin.

4. Secure your 'Secure Shell' (SSH) Access

I strongly recommend running SSH on a separate IP and disallowing direct login via 'root' (that's why we added an admin user in step 2). So fire up your editor and load the ssh config file /etc/ssh/sshd_config.

  • Uncomment #Protocol 1,2 and change it to Protocol 2
  • Bind SSH to a separate IP addressUncomment and set ListenAddress xxx.xxx.xxx.xxx where xxx's represent your non-primary IP address. Most providers will allocate 1-4 IP's to every account, if not you can request an IP for a small fee (usually) or if you don't want to run SSH on a seprate IP, at least bind it do a different port. Warning: before you bind your SSH service to the new IP, make sure you can access your server via the IP (check /sbin/ifconfig and try ping ip)
  • Uncomment PermitRootLogin yes and change it to "no" - This will force you to login via the admin user and su - to get root
  • I usually also add/uncomment: KeepAlive yes and Compression yes - but that's up to you!
  • Restart your SSH service:
$ /sbin/service sshd restart
- ssh into old ip
- connection should fail!
- ssh into new ip
- try logging in as 'root'
- it should fail!
- ssh into new ip
- try logging in as 'admin' - voila!
- '_su -_' to get root

5. Put a scarecrow on your front yard

Fire up your editor and open /etc/motd - anything you store in this file will be shown to every user that accesses your server via SSH. I use a standard text: This computer system is for authorized users only. All activity is logged and regulary checked by systems personal. Individuals using this system without authority or in excess of their authority are subject to having all their services revoked. Any illegal services run by user or attempts to take down this server or its services will be reported to local law enforcement, and said user will be punished to the full extent of the law. Anyone using this system consents to these terms.

6. Disable Telnet

Test if you have telnet running (telnet yourip). Most new servers have it disabled, but if your's doesn't, open up /etc/xinetd.d/telnet and change disable = yes to disable = no. Save the file and restart xinetd (/etc/init.d/xinetd.d restart)

7. Install a firewall

Install APF Firewall, or some other pet peeve of yours that will do the job. There is a number of great tutorials for configuring APF, take a look here or here. I always enable AD (Anti-Dos), Egress Filtering (Out), DShield (checks suspicious IP's). All these features add some overhead, but I tend to err on the side of too much security (Hey, at least I sleep well!).

8. Add Brute-Force detection

If you chose APF, you're in luck, BFD (Brute Force Detection) is a great add-on for APF which will scan your system log (/var/log/messages) for any IP that is trying to brute force into your system and once found, will call on APF to ban the user and deny him access to the server altogether. (I get 1-5 of these everyday on my production servers!) Installation is very easy:

$ wget http://www.r-fx.ca/downloads/bfd-current.tar.gz
$ tar -zxvf bfd
$ cd bfd
$ ./install.sh
$ vim /etc/bfd/conf.bfd

That's it! When you ran install.sh a cronjob was created which will check your logs every 5 minutes and ban all brute-force attempts!

9. Install a socket monitor

LSM (Linux Socket Monitor) is a very useful little program - it monitors open sockets on your server and notifies you when new ones are created. This is extremely useful because once you server goes live you rarely if ever open any new ports, so if a rogue process opens a new port to the outer world... we will know right away! Installation is virtually identical to BFD:

$ wget http://www.r-fx.ca/downloads/lsm-current.tar.gz
$ tar -zxvf lsm
$ cd lsm
$ ./install.sh
$ vim /usr/local/lsm/conf.lsm
$ ./usr/local/lsm/lsm -g

This will generate a 'base file' to compare open sockets to - you should run this after you have configured all your service. Again, the installer created a cronjob for us which will check all open socket to the base file. Moving on...

10. Add a paranoid security layer

LES (Linux Environment Security - Non VPS/Virtuozzo Users) is for the really paranoid (like myself), this will make most of your configuration and executables either immutable or non-writable. This does tend to break automatic updates, but I think it's worth the trouble. Unfortunately, Virtuozzo does not support chattr so most of the functionality is missing, but if you're on Xen or your own dedicated server I strong recommend using LES. Installation is identical to LSM/BFD.

11. Secure your tmp directory

This is a little more involved, but a must do for server security. Most URL exploits rely on the fact that they can download a program/script to your tmp directory and execute it from there. By default the /tmp mount will allow this, so instead, we will create a new /tmp folder with noexec and nosuid set on the partition. Check out this guide for dedicated servers and this guide for virtual servers.

12. Harden your Sysctl parameters

By default your server will not check for ACK spoofing and in general be very liberal with its timeouts - making you a prime target for DDoS and SYN attacks. We can fix that by using a script here, or if you're curious and want to customize it exactly to your needs click here, here or here.

13. Secure PHP

If Apache is still on your list of required services, chances are it will have php enabled, so let's add a layer of security there. By default, php will allow the user to call most of the kernel methods on your system - disabling most of these is highly recommended but it can also break the functionality of some php programs, so it all depends on what you're running. Do locate php.ini to find the location of your php configuration file and then fire up your editor and look for disable_functions = ... within the file. I usually use: disable_functions = "exec, system, passthru, readfile, shell_exec, escapeshellarg, escapeshellcmd, proc_close, proc_open, ini_alter, dl, popen, parse_ini_file, show_source, curl_exec". Close the file and restart your apache.

14. Security through obscurity

If someone is seriously trying to break into your system, this usually wont stop them, but for 95% of the cases where automated bots are simply scanning networks for old and exploitable services this can be a great feature. We're simply going to hide the version numbers of the services that are accessible from the outside:

  • Apache - open your config file and change ServerSignature on to ServerSignature off and right below add ServerTokens prod. Restart your apache and now it will simply return 'Apache' in the header instead of (Apache 1.3.x .. list of all the loaded modules).
  • Repeat same steps for other services. (Google is your friend here)

15. Install RootkitHunter

I hope you will never actually find this app to be useful, because if you do, chances are your server is compromised. But it's a must have tool nonetheless, it will scan and compare md5 hashes and filesizes on most of critical executables to ensure that no one has tampered with your server. Check out the following guides for installation: here, here or here. chkrootkit is another useful tool, in fact I usually run both on all my servers (guide here).

16. Reward yourself with a _____ if you made it to here

If you've done all of the above, I'm fairly confident that you've substituted your Swiss cheese setup for a concrete wall. This is by no means a complete guide, there are dozens of extra things you could (and probably should) do: install system monitoring services (HotSanic, Monit, etc), secure apache with additional modules (mod_security and mod_evasive are great addon's, albeit at 50% decrease in your throughoutput), use SSH keys instead of logins, test your Rails application, etc.

Learn and practice the process

Security is a process, not a product or a service - anyone that claims that they deliver a 'secure' server to you is lying through their teeth. You can spend days on securing your environment, but if one of your developers decides to put a sticky note on his monitor with his login and password... You're as weak as your weakest link, and that usually means the users, not the system. If you're interested, there is a great high-level book on the subject by Bruce Schneier (Secrets and Lies: Digital Security in a Networked World).

Ilya GrigorikIlya Grigorik is a web ecosystem engineer, author of High Performance Browser Networking (O'Reilly), and Principal Engineer at Shopify — follow on Twitter.