Server Security is a vast topic, and there are many setps and procedures, tools to secure the server.

If you need help with setting up any of the following please do mail me at admin@serversignature.com

Server Security in Brief

Setup the firewall first before installing any softwares:  A firewall should be the first thing installed - APF or CSF http://rfxnetworks.net/apf.php or http://www.configserver.com/

Enable Email Alert on root login

cd /root ; vi  .bashrc
Scroll to the end of the file then add the following:
echo 'ALERT - Root Shell Access (YourserverName) on:' `date` `who` | mail -s "Alert: Root Access from `who | cut -d"(" -f2 | cut -d")" -f1`" admin@serversignature.com

Install BDF

* If you have APF already then install brute force monitor (BFD) also by rfxnetworks. BFD will monitor your ssh and ftp services and automatically ban users that try to brute force a password.

Install PMON

* Install socket monitor (PMON). This tool will alert you whenever a new port is opened on the server. This is very helpful in detecting any users running weird processes or attempting to run backdoors. When any program that it does not recognized is started it will email you with the information. http://rfxnetworks.net/pmon.php

Install SIM

* Install system integrity monitor (SIM) which is also by rfxnetworks. SIM will automatically detect when a service is down and restarts it. http://rfxnetworks.net/sim.php

OFF compilers

* Turn off compilers. Most rootkits come precompiled but not all of them do. It will also prevent shell users from trying to compile any irc related programs. To turn the compilers on switch the off to on. /scripts/compilers off

Disable ping access to the server.

vi  /etc/sysctl.conf

net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_echo_ignore_all = 1

Password Options

* edit /etc/login.defs to configure many password options on your system.

Test Password

* Test password using - http://www.openwall.com/john/

Check password Strength

* install tools like pam_passwdqc to check the strength of passwords.

Enable SSH keys

* Enable public key authentication for SSH and disable password authentication read more >>

Change SSH ports

* Move SSH access to a different port. People are looking for port 22 as a possible way to access your servers.

* You can modify the port that SSH runs on within /etc/ssh/sshd_config. Change the line that says #Port 22 to a different port such as: Port 1653.

* Use SSHv2 only as SSHv1 is not secure. Make sure to change the line in /etc/ssh/sshd_config that says #Protocol 2,1 to Protocol 2.

Set Limit

* Set Shell Resource Limits for you users to prevent applications and scripts from using all up your resources and taking down your server. You can configure shell resource limits in /etc/security/limits.conf on most Linux systems.

* One of the best tools for preventing malicious Apache use is mod_security. This can be installed in Addon Modules in the cPanel section of WebHost Manager. You can find information about mod_security at http://www.modsecurity.org/.

mod_security

--
First we will download and unzip mod_security. This guide compiles for apache1.3.x which is what cPanel currently uses.
#wget http://www.modsecurity.org/download/...y-1.8.4.tar.gz
#tar zxf mod_security-1.8.4.tar.gz
#cd mod_security-1.8.4/apache1

Next compile mod_security at a module:
#/etc/httpd//bin/apxs -cia mod_security.c

Make a backup of your httpd.conf before touching anything so you have something to go back to if it does not work.
#cp /etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf-mod_sec

Now edit the httpd.conf
pico -w /etc/httpd/conf/httpd.conf

Scroll down below the following line:
AddModule mod_security.c
The rules listed in the text file below can just be pasted in. They are a collection of rules, many of them taken from snort, that block most of the common attacks while still letting normal requests by.
http://eth0.us/faq/modsec.txt

Create the error log file:
#touch /var/log/httpd/audit_log

Restart apache
#service httpd restart

If sites start to have problems look at error log.
/var/log/httpd/audit_log

--

* When compiling Apache, you should include suexec to ensure that CGI applications and scripts run as the user that owns / executes them.

* Cpanel also recommend compiling Apache + PHP with PHPsuexec. PHPsuexec forces all PHP scripts to run as the user who owns the script. This means that you will be able to identify the owner of all PHP scripts running on your server. If one is malicious, you will be able to find it’s owner quickly and resolve the issue. To compile Apache + PHP with PHPsuexec, select the PHPSuexec option in the Apache Upgrade interface in WHM or when running /scripts/easyapache from the command line.

* Enable PHP’s open_basedir protection. This protection will prevent users from open files outside of their home directory with PHP. This can be enabled in Tweak Security within WebHost Manager.

Disable PHP Functions

Search the php.ini file for: disable_functions =  and Add the following:
disable_functions = dl,system,exec,passthru,shell_exec

Enable PHP open_basedir Protection

PHP's open_basedir protection prevents users from opening files outside of their home directory with php. This security tweak uses Apache DSO style directives. If PHP is configured to run as a CGI, SuPHP or FastCGI process, the open_basedir setting must be manually specified in the relevant php.ini file.

* Include safe_mode for PHP 5.x and below. Safe_mode ensures that the owner of a PHP script matches the owner of any files to be operated on. You can enable safe_mode by changing the safe_mode = line in php.ini to safe_mode = On.

* Enabling suEXEC provides support for Apache to run CGI programs as the user ID of the account owner. suEXEC is not PHPSuExec. Please refer to the documentation for a detailed explanation of how each handler functions.

* Use a separate partition for /tmp that is mounted with nosetuid. Nosetuid will force a process to run with the privileges of it’s executor. You may also wish to mount /tmp with noexec after installing cPanel. Check the mount man page for more information. Also, Running /scripts/securetmp will mount your /tmp partition to a temporary file for extra security.

* Move mails to maildir format - using /scripts/convert2maildir. Make sure to back up your current mail before converting to maildir, this can be done within /scripts/convert2maildir. If you see maildir is enabled when running /scripts/convert2maildir, you are already using maildir, and will not need to convert.

* Users do not require the use of C and C++ compilers. You can use the Compilers Tweak within Tweak Security in WebHost Manager to turn off use of the compilers for all unprivileged users, or to disable them for specific users only.

* netstat -anp : Look for programs attached to ports that you did not install / authorize.

* find / \( -perm -a+w \) ! -type l >> world_writable.txt : Look at world_writable.txt to see all world writable files and directories. This will reveal locations where an attacker can store files on your system.

* find / -nouser -o -nogroup >> no_owner.txt : Look at no_owner for all files that do not have a user or group associated with them. All files should be owned by a specific user or group to restrict access to them.

* Monitor your system and to detect rootkits, backdoors, etc. Here are some commonly available utilities:

• Tripwire - Monitors checksums of files and reports changes.
  http://tripwire.com or http://sourceforge.net/projects/tripwire
• Chrookit - Scans for common rootkits, backdoors, etc.
  http://www.chkrootkit.org
• Rkhunter - Scans for common rootkits, backdoors, etc.
  http://www.rootkit.nl/projects/rootkit_hunter.html
• Logwatch - Monitors and reports on daily system activity.
  http://logwatch.org

/TMP patition

* The /tmp partition is one the common places for script kiddies and crackers alike to place trojans or scripts. Because of that you should have the /tmp partition mounted noexec. First we need to check if your /tmp is secure.
#df -h |grep tmp

If that displays nothing then go below to create a tmp partition. If you do have a tmp partition you need to see if it mounted with noexec.
#cat /etc/fstab |grep tmp

If there is a line that includes /tmp and noexec then it is already mounted as non-executable. You will also want to check if /var/tmp is linked to /tmp.
ls -alh /var/ |grep tmp

If it shows something to the effect of "tmp -> /tmp/" then you are ok. If not go ahead an remove the old /var/tmp and replace it with a sym link to /tmp.
#rm -rf /var/tmp/
#ln -s /tmp/ /var/

/tmp

If you do not have any /tmp partition you will need to follow the directions below to create and mount a partition.

Create a 190Mb partition
#cd /dev/; dd if=/dev/zero of=tmpMnt bs=1024 count=200000

Format the partion
#mke2fs /dev/tmpMnt

Make a backup of the old data
#cp -Rp /tmp /tmp_backup

Mount the temp filesystem
#mount -o loop,noexec,nosuid,rw /dev/tmpMnt /tmp

Set the permissions
#chmod 0777 /tmp

Copy the old files back
#cp -Rp /tmp_backup/* /tmp/

Once you do that go ahead and start mysql and make sure it works ok. If it does you can add this line to the bottom of the /etc/fstab to automatically have it mounted:
/dev/tmpMnt /tmp ext2 loop,noexec,nosuid,rw 0 0

Next delete the old /var/tmp and create a link to /tmp
#rm -rf /var/tmp/
#ln -s /tmp/ /var/

If everything still works fine you can go ahead and delete the /tmp_backup directory.
#rm -rf /tmp_backup

CHANGE PERMISSION

#chmod 700 /usr/bin/rcp
#chmod 700 /usr/bin/wget
#chmod 700 /usr/bin/lynx
#chmod 700 /usr/bin/links
#chmod 700 /usr/bin/scp

Install rkhunter

Rootkit scanner is scanning tool to ensure you for about 99.9%* you're clean of nasty tools. This tool scans for rootkits, backdoors and local exploits by running tests like:

http://sourceforge.net/projects/rkhunter/

Download and unzip rkhunter
#cd /usr/local/src/
#wget http://downloads.sourceforge.net/project/rkhunter/rkhunter/1.3.6/rkhunter-1.3.6.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Frkhunter%2F&ts=1289196532&use_mirror=biznetnetworks
#tar -zxf rkhunter-*
#cd rkhunter

Install it
#./install.sh --install

Now create a cronjob so it will email you with notifications to the root mailbox:
#crontab -e

At the bottom add the following line
16 0 * * * /usr/local/bin/rkhunter -c --nocolors --cronjob --report-mode --createlogfile --skip-keypress --quiet

# ./rkhunter  -c
[ Rootkit Hunter version 1.3.6 ]

Checking system commands...

Performing 'strings' command checks
Checking 'strings' command                               [ OK ]

Performing 'shared libraries' checks
Checking for preloading variables                       [ None found ]
Checking for preloaded libraries                         [ None found ]
Checking LD_LIBRARY_PATH variable                   [ Not found ]

Add to cron

/etc/cron.daily/rkhunter.sh

add the following replacing your email address:

#!/bin/bash
(/usr/local/bin/rkhunter -c --cronjob 2>&1 | mail -s "Daily Rkhunter Scan Report" admin@serversignature.com)

chmod +x /etc/cron.daily/rkhunter.sh

Updating rkhunter
gets the latest database updates from their central server and matches your OS better to prevent false positives.

rkhunter --update