CU Home
Columbia University Information Technology
Server setup and maintenance in the Columbia University network environment

Server setup and maintenance in the Columbia University network environment

The following paper describes recommended practices that can be used to create and/or maintain a network-attached server that will provide both a functional environment and security for its data. Since it is impossible to have both a fully secured environment and useful network connectivity, there will always be a balance between security and functionality. It is possible to run a reasonably secure server, although it takes work and vigilance on the part of the system administrator. A server, if not properly maintained, has the same vulnerabilities as a desktop system and more. All of the rules for the security of desktop systems apply (see http://security.columbia.edu), plus the additional considerations below.

This paper is meant to be a living document on how to create and maintain a usable server securely. If you feel that there are corrections or additions needed, please contact joel@columbia.edu

Rule 1 - Don't put everything on one server.

With the advent of inexpensive computers and very fast network connections, the incentive to put all of the services on one system has been greatly reduced. While it may seem to be a reasonable way to reduce costs, from a security perspective it really does not make sense. Putting each service on single purpose machines allows you to secure each machine more tightly and reduces the risk of one break-in compromising all of your data and services. Additionally, by separating your services, you reduce the risk of a program check in one service taking down another un-related service on the same machine.

When choosing a password for the administrator ID on a server, make sure that you do not use the same password as from any other ID that you may have. Additionally, make sure that you use a different password for every server.

This rule may have to be broken occasionally for performance reasons. An example of this would be a front-end application that is designed to run on the same server as the database. This rule should only be broken if absolutely necessary.

Good planning is essential; spend the time upfront and create a plan before going out and investing in hardware.

Rule 2 - Keep servers physically secure.

Make sure that you have a secure environment for your machines. Keep your servers in a cool, clean and dry room with a good lock on a strong door. The key to that room should be protected and monitored. Machines should never be kept on the floor - always have the computer raised at least 4 inches. You should also make sure that your server is protected from water coming from above. Purchase a UPS system for each server that can shut the machine down properly in case of a power outage.

Servers should never be used by anyone as a desktop machine, even briefly to check email or surf the web. One wrong click and the machine has been compromised. It is impossible to protect a machine from a user who has access to the primary console. A server should be dedicated to serving only. Physical access to servers should be limited to personnel that have legitimate reasons, and an access logbook should be kept. Remember, if someone can get physical access to the machine, they can get logical access to all data stored by booting the machine from a CD or other media.

Rule 3 - Check the security of all applications on the server.

When using a vendor's special purpose application, make sure that you have an extensive conversation with the vendor about the security of their product. It is equally important to check that all of your current applications have been updated to meet the demands of the current security environment. Many legacy applications were written before security was as big an issue as it is today. Make sure that the vendor is aware that their product will be exposed to the traffic on the Internet and will not be behind a corporate network firewall.

Get a written statement, if possible, from the vendor as to the security of the product and make sure that you install the latest security patches available and keep them up to date. Find out how patches are announced and distributed for any product that is installed on the server. It is possible that a security hole in one product may allow access to the entire server. If a product cannot be secured, consider looking for a replacement or look for a way to encapsulate the insecure product within a secure shell.

If it is a third-party turnkey system, the contract should state who is responsible to updating the operating system and associated software on that server. Again, the vendor should be aware that the systems will be given a live address directly on the Internet and that there are no firewalls protecting this machine from attack. Speak to the vendor about automatic updates, anti-virus, PestPatrol and either turning on or installing a software firewall on the server.

Occasionally, you may run into a situation where an application cannot be secured and there is no replacement available, contact security@columbia.edu for help in providing a secure way to run these applications.

Rule 4 - Get on mailing lists and keep up to date.

In security, information is power. The bad guys depend on being ahead of the curve by having the latest compromise available and hoping that you do not have the patch installed yet. Being on several security and product mailing lists may make your email busy, but it may also save you hours of grief. Some good lists are: www.cert.org, www.sans.org, www.microsoft.com/security/default.mspx (for Microsoft users). The important thing to remember is that you need to act on the information ASAP. The main Columbia University security list you can get on as systems administrators, SecurityFolks, can be subscribed to by just sending an email to securityfolks-subscribe@lists.columbia.edu. Subject and text are ignored, just reply to the confirmation message.

Rule 5 - Turn off unneeded services.

Any service that runs on a machine and accesses the network is a potential security hole. Operating systems contain millions of lines of code, and an error in any one of them can lead to a compromise. Most security compromises rely on a technique referred to as a buffer overrun. In this type of attack, a string of characters is sent to a service port that exceeds the length of the buffer that was allocated to receive the message. The string has been carefully crafted to create an error condition in the program by overwriting some code in the server.

Rule 6 - Use anti-virus and firewall software.

A properly configured firewall will help you recover from problems not covered by rule number 4. The purpose of a firewall is to block access by network traffic that is trying to access ports on your server that are not expecting traffic.  Windows, Linux and Mac servers have firewall software built in, though it will still need to be activated and properly configured.

Anti-virus software serves a different purpose: it uses software signatures of various malicious pieces of code to detect if that code is present in files on your server. Once detected, the anti-virus program can disable and remove the malware - or at least tell you that it found a problem.

Other software you should consider is an anti-spyware product like PestPatrol. This product monitors your system for the presence of Trojans, keyloggers and other non-virus compromises. Additionally, Columbia University has a site license for the Tripwire product. This software, when properly set up, can detect any unauthorized change in the programs running on the machine. Using Tripwire on an evolving server is labor intensive; since it needs to be updated any time there is a change to a program being monitored. If you are interested in using Tripwire, send email to security@columbia.edu, we will arrange to get you the product and a license key. The product is available for both Unix and Windows.

Rule 7 - Keep logs, and keep them safe.

Think of logs as a form of insurance. Insurance will not help prevent an accident, but it will help you recover from one. Logs are one of the ways you can figure out what happened. Logging should be configured to keep track of access (both success and failure) to any important data that is stored. You should also log all access to the system, both success and failure. Check with the vendor of any software you are running to find out what logging options are available.

The logs should be kept in a safe place on your servers - a really good suggestion is to change the default location that the logs are stored (the bad guys have the same programs that you do) and keep a set of dummy logs in the default location (a copy of the real log will work nicely.) A savvy bad guy will modify the log files to conceal their attack on your server; they may miss the fact that you have moved the real log to a more secure place. Having logs will help you figure out what happened, but only if they are read in a timely fashion.

A part of the daily maintenance of any server (especially one containing important information) is to look through the logs for any unusual activity. If you discover a compromise, please see Rule 13.

Rule 8 - Keep track of what's running.

One of the most important pieces of information you need to keep about your server is what kind of information is being stored or accessed by that server. All information is important to someone and can be put in two classes, public and private - in general, any private information that became public or got stolen could compromise someone's privacy or identity, or embarrass Columbia University. If your server contains any private information, you should pay special attention to Rule 13.

Keep a list of all of your servers and the software that is running on them. Include software, versions, keys, and support numbers. You should also look at the task list (use the task manager and look at processes) and be familiar with the various tasks that your system runs. You should be able to look at your server's list of running programs and tell if something doesn't look right. If a root kit has been installed, you may not see the bogus tasks running, but simple "script kiddy" hacks will be apparent.

Physically label the machine with the hostname and the registered IP address; this will help you locate the machine in case of an incident or problem (Note: at Columbia, we ask that all machines, even servers, use DHCP. If you register your host name at http://www.columbia.edu/acis/access/dns/, you will always get the same IP address that you were assigned.)

It is generally a good idea not to give a server a hostname that gives away it's purpose. For example, you should not name your sql server sql.dept.columbia.edu. Why make it easier for the bad guys?

Rule 9 - Use hardening tools to limit security holes.

Hardening tools are programs or scripts that can be used to configure a server to make it much harder for someone to compromise the machine. They do this by changing access lists, checking passwords, getting rid of sample code, getting rid of default IDs and in general, making sure that your machine is set up differently than the one that comes out of the box. Making a system sufficiently different from the standard will make it difficult for someone to figure out where things are stored and what they would have to do to change them. The longer it takes to compromise your server, the better chance there is that someone will notice the attempt and stop it before they break in. Good references for hardening programs are at: www.nsa.gov/snac/ and http://csrc.nist.gov/checklists.

Rule 10 - Do not allow insecure or unencrypted remote access.

Do not enable services that require the use of clear text passwords. Any password that is transmitted over the Internet in clear text can be intercepted and used. It is human nature to use the same password for more than one system. Chances are if your password is stolen, it will give the thief access to many of your machines - it is important to conquer human nature and use different passwords. Make sure that all passwords used by people that have access to the server are strong passwords. If you need to use a remote access service, make sure that the service supports SSH login. You should also avoid remote administrator login. If that is not possible, use a secure VNC product and tunnel the connection over SSH.

Remember to examine the security of the remote system that you are using to do your remote logins. A remote machine with a keylogger installed will expose your server's passwords, as will a remote machine with a Trojaned SSH program. Any machine that you use for remote access to a server must be as secure as the server, this means that your home machine should be fully secured.

If your home machine is the family computer, used by kids and others for entertainment, you can assume that the machine is not secure.

Rule 11 - Update update update.

All software, whether vendor applications or operating systems, needs to be checked on at least once a week for the availability of security updates and patches. Make sure that you are plugged into information resources that will inform you that updates are available. You should check frequently with the vendor of any software that you run and get on the mailing list for updates. If the vendor does not support a list, check with various user groups to see if there is a public list you can subscribe to. This is especially important for systems that are being supplied by the vendor. Turnkey systems frequently are designed to be used in a "secure" environment, not the open environment that exists at Columbia University. Make sure that all vendors are aware that there is no corporate firewall protecting their machine and application.

Rule 12 - Make a recovery image and do Backups.

When your server is securely configured, make an image of the system. This will allow you to recover from a hardware failure or a system compromise. Make sure that your image is kept up to date.

A DVD writer is now cheap enough; you can also use it to store your system image. You should update your image files whenever the server is updated or reconfigured.

You should backup your data regularly. Current backups are the quickest way to get a server back online after a rebuild because of a compromise or hardware failure. Backups should be stored in a different location than the server. If there is a physical break in and the hardware is stolen, you do not want the backups going with the server, also, in case of fire, you don't want the backups damaged.

Anyone running a critical server should have a detailed recovery plan in case of failure. Think about what would happen if your server where to be down for a day, a week or even a month! You should test this plan to make sure that the details of the recovery match the reality of how the server is being maintained, i.e. backups are being done and can be read, image disks exist and can be restored, etc. If you server is especially critical, you should consider having duplicate hardware.

We are running a backup service using the Connected backup software. This software encrypts the backups over the wire as well as on the server. Please contact Joyce Fetterman for information about using the Connected backup system to securely back up your data.

Rule 13 - Be prepared in case the worst happens.

A server by definition is critical to someone. You can always count on Murphy (http://www.murphys-laws.com/) to make sure that your server will need attention as soon as you are not available.

The university network is monitored for serious problems such as botted machines, DOSing machines, and scanning machines and in general, any system that is either using unusual amounts of network resources or is communicating with known compromised systems. If your server is compromised in a way that our network monitoring software detects, you will be notified of the problem.

Please make sure that we have contact information for the person or persons responsible for your servers - this information should include email and voice/cell numbers, including off-hour contact. Please send your contact information to security@columbia.edu. If we cannot contact the person responsible for a compromised server, we will cut the network connection as close as possible to the machine.

As stated in Rule 8, if a machine with private information is compromised, there could be serious consequences as discussed in the paper on incident reporting procedures.

If you discover a compromise, remove the system from the network immediately, then report it to security@columbia.edu as soon as possible.

CUIT Security team
Joel Rosenblatt
August 31, 2005

Here are some good references - Note that quotes are directly from the web pages:

http://www.educause.edu/Search/644 - Use this search page to find information on your topic of concern. EDUCAUSE is a nonprofit association whose mission is to advance higher education by promoting the intelligent use of information technology.

www.us-cert.gov - The United States Computer Emergency Readiness Team (US-CERT) is a partnership between the Department of Homeland Security and the public and private sectors. Established in 2003 to protect the nation's Internet infrastructure, US-CERT coordinates defense against and responses to cyber attacks across the nation.

www.cert.org - Established in 1988, the CERT Coordination Center (CERT/CC) is a center of Internet security expertise located at the Software Engineering Institute, a federally funded research and development center operated by Carnegie Mellon University.

www.first.org - FIRST is the premier organization and recognized global leader in incident response. Membership in FIRST enables incident response teams to more effectively respond to security incidents - reactive as well as proactive.

www.cyberpartnership.org/CommonSenseGuideBus.pdf - Small business best practices guide. There are many aspects of this document that can be applied to servers in educational institutions.

www.cerias.purdue.edu/tools_and_resources - The Center for Education and Research in Information Assurance and Security (CERIAS) is currently viewed as one of the world's leading centers for research and education in areas of information security that are crucial to the protection of critical computing and communication infrastructure. CERIAS is unique among such national centers in its multidisciplinary approach to the problems, ranging from purely technical issues (e.g., intrusion detection, network security, etc) to ethical, legal, educational, communicational, linguistic, and economic issues, and the subtle interactions and dependencies among them.

www.csrc.nist.gov/pcig/ppsp.html - This site contains security practices from academia, public, and private organizations.

www.cve.mitre.org/about/ - Common Vulnerabilities and Exposures (CVE) is a list of standardized names for vulnerabilities and other information security exposures - CVE aims to standardize the names for all publicly known vulnerabilities and security exposures.

www.nsa.gov/snac/os/win2k/w2k_group_policy_ref.pdf - This manual is not a how-to guide for using Group Policy in a secure configuration, but more a map to help the reader locate specific policies within the Group Policy Snap-in for a given Active Directory container. It is hoped that this map will alleviate some of the complexity in managing and understanding Group Policy.



Reporting Security Problems

Send reports of security incidents, attacks, or questions to security@columbia.edu