Server Installation

From MyWiki

Contents 1 Dell Poweredge 1950 2 BIOS 3 Linux Install 4 inittab 5 useradd 6 Local Passwords 7 SUDOers 8 SELinux 9 Networking 10 Disabling IPv6 11 iptables 12 NTP 13 SSHD 14 Kerberos 15 Yum (Fedora or Centos) 15.1 For Xen Servers 15.2 For All Servers 16 up2date (RHEL) 17 cron-apt (Debian) 18 Sendmail 19 Root Emails 20 Locate 21 Install vmware-tools 22 Services 23 Verification 24 DNS Registration 25 Sun Java and GNU Java 26 Backup 27 X Windows

Dell Poweredge 1950

If you're installing on a Dell Poweredge 1950, there is a bug in the IPMI management portion of the firmware on the integrated Broadcom NICs. Do not pass go and do not collect $200 before following the instructions on this page.

Since the default Broadcom dual NIC firmware has been upgraded, this bug is no longer relevant.

BIOS

• Set HW clock to UTC (+5 in summer, +6 in winter in the central time zone)
• Set Virtualization Technology to Enabled
• Optional: disable 2nd network adapter if two are present on the motherboard (such as in the PE1950).

Linux Install

• Boot from CD using FC7 or FC8 install CD (need the first 4 CDs from the full 5 CD set OR you can use just the first CD with a network install)
• For a Red Hat EL 4 installation you will need the first four of the five CDs downloaded via subcription from redhat.com ($60 per year, no support, http://www.redhat.com/solutions/education/academic/individual )
• "linux text" install OR 'linux text askmethod'

The latter allows you to install from a network repository. A fast mirror to use is Argonne's HTTP or FTP download site:

 mirror.mcs.anl.gov

then choose the desired version, e.g., one of:

 pub/fedora/linux/releases/7/Everything/x86_64/os
 pub/centos/5.1/os/x86_64/
 pub/fedora/linux/releases/8/Everything/x86_64/os/
 pub/fedora/linux/releases/8/Everything/i386/os/ 

Also, if you leave out the 'text', fedora will start a GUI install, which is nearly the same except for the software selection choices.

• If you're installing into a physical servers (i.e. not a VM):
  • If SCSI-style HDs, you should see a message indicating that it is loading "mptbase" and "mptsas".
  • "Remove all partitions on selected drives and create default layout."

  VG VolGroup00    476640M VolGroup  
  LV LogVol01        1984M swap
  LV LogVol00      474656M ext3       /
  /dev/sda
    sda1  1     13    101M ext3       /boot
    sda2 14  30394 238315M physical v
  /dev/sdb
    sdb1  1  30394 238417M physical v

• If you're installing into a virtual server, do not use SCSI (why?) or LVM (why?).
  • If you're virtual drive is SCSI, stop this procedure now, delete the SCSI drive, and create a new virtual IDE drive, then resume here.
  • Do not partition the IDE hard drive by using the default layout. Instead, create the following partitions:

  #fdisk /dev/sda
   
               Device Boot  Start   End   Blocks  Id  System
  (primary) /dev/sdb1   *       1    25   (200M)  83  Linux 
  (primary) /dev/sdb2          26   150  (1024M)  82  Linux swap
  (primary) /dev/sdb3         151  2610    (all)  83  Linux

• Use GRUB
• Boot Loader Configuration
  • Label: Fedora Core
  • Device: /dev/VolGroup00/LogVol00
  • Where: /dev/sda Master Boot Record (MBR)
• Ethernet
  • Use DHCP
  • Hostname: xxxxxxxx.ncsa.uiuc.edu
• Firewall
  • Customize
  • Enable SSH
• Security Enhanced Linux
  • Disabled
• Timezone
  • System clock is UTC
  • America/Chicago
• Root password
  • Be sure to update passwords.gpg
• Package Selection
  • Uncheck all three (Office, Devel, Web Server)
  • Select "Customize Software Selection Now"
  • Uncheck all package groups, except (for Red Hat v4.4, text install):

 X Window System
 GNOME Desktop Environment
 Editors (add Emacs & Xemacs)
 Graphical Internet (Hit F2, then firefox only)
 Text-Based Internet
 Administration Tools
 Compatibility Arch Support

• Uncheck all package groups, except (for Fedora Core 6, GUI install):

 Desktop Environments
   GNOME
 Applications
   Editors (add Emacs)
   Graphical Internet (Firefox only)
   Text-Based Internet (add lynx)
 Development
   as is
 Servers
   Uncheck Printing Support
 Base System
   as is
 Languages
   as is

• Click Continue to install packages.
• Reboot from HD

inittab

• Verify /etc/inittab is set to runlevel 3.

 id:3:initdefault:

• If the install is for a guest under VMware, it can be useful to comment out the following line so that the guest will not shutdown when you hit cntl-alt-del to lock your workstation.

 ca::ctrlaltdel:/sbin/shutdown -t3 -r now

useradd

• Add user accounts as necessary.

 useradd -u nnnnn UserName

If you want to have consistent NCSA UIDs, you can find them by: grep USERNAME /afs/ncsa/common/etc/passwd .

Local Passwords

Remove local passwords if necessary and appropriate. See [1] for additional details.

How this is done varies from OS to OS. However the following rule is believed to work on all relevant systems.

This can be done by editing the /etc/shadow file and replace the user's (hashed) password with a short string (less than 13 characters) that does not contain the characters ! * or $.

For example:

# grep globus /etc/shadow
globus:NOPASS:14004:0:99999:7:::

SUDOers

• Edit /etc/profile

 EDITOR=/usr/bin/nano
 export EDITOR

• Use command visudo to edit the sudoers file

 User_Alias POWER_USERS = user1, user2, ..., userX
 ...
 POWER_USERS ALL=(ALL) NOPASSWD: ALL

• While still in edit on the sudoers file, comment out the following line:

 #Defaults    requiretty

Related background on this setting: I want to run pdsh (parallel distributed shell) to run scripts on many servers concurrently. The network connections are ssh to the target servers, authenticating me with my normal user account via ssh keys. The shell script on the target server is then executed, but no tty is associated with the running script. When the script encounters a sudo command, failure occurs, with the message:

 "sudo: sorry, you must have a tty to run sudo"

There exists a configuration parameter in /etc/sudoers called requiretty, defined as:

requiretty  If set, sudo will only run when the user is logged in to a
                real tty.  This will disallow things like "rsh somehost
                sudo ls" since rsh(1) does not allocate a tty.  Because it
                is not possible to turn off echo when there is no tty
                present, some sites may with to set this flag to prevent a
                user from entering a visible password.  This flag is off by
                default.

BTW, yes, I tried setting PDSH_SSH_ARGS_APPEND="-t", and that didn't work.

SELinux

Disable SELinux

• edit /etc/sysconfig/selinux and set

 SELINUX=disabled

• Reboot
• After reboot, this command should yield the response "Disabled"

 getenforce

Networking

• Create a host DNS entry using the NCSA Network Engineering website.
• If you get a "no link present" error on the adapter when starting the network, you may need to add the following snippet to /etc/sysconfig/network-scripts/ipcfg-eth0

check_link_down() {
  return 1;
}

• Check /etc/sysconfig/network

 HOSTNAME=<your fqdn>

• Check /etc/hosts
• Check /etc/resolv.conf

Disabling IPv6

• If you want to disable IPv6, add the following line to the /etc/modprobe.conf file:

alias net-pf-10 off
alias ipv6 off

• Paste into /etc/sysconfig/network:

NETWORKING_IPV6=no

• Stop ip6tables firewall and prevent its startup in the future:

/sbin/service ip6tables stop
/sbin/chkconfig ip6tables off

• Paste into /etc/sysconfig/network-scripts/ifcfg-eth0:

IPV6INIT=no

iptables

Note: A default install of FC5 should have set up the correct rules already.

Allow only SSH (as a default starting point at least). Edit the file /etc/sysconfig/iptables to add the line:

 -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

then use this command to have the iptables kernel module reread its configuration:

 service iptables restart

You can check the iptable configuration using:

 iptables-save 

which should result in something similar to

 *filter
 :INPUT ACCEPT [0:0]
 :FORWARD ACCEPT [0:0]
 :OUTPUT ACCEPT [611:69223]
 :RH-Firewall-1-INPUT - [0:0]
 -A INPUT -j RH-Firewall-1-INPUT 
 -A FORWARD -j RH-Firewall-1-INPUT 
 -A RH-Firewall-1-INPUT -i lo -j ACCEPT 
 -A RH-Firewall-1-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT 
 -A RH-Firewall-1-INPUT -p ipv6-crypt -j ACCEPT 
 -A RH-Firewall-1-INPUT -p ipv6-auth -j ACCEPT 
 -A RH-Firewall-1-INPUT -d 224.0.0.251 -p udp -m udp --dport 5353 -j ACCEPT 
 -A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT 
 -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT 
 -A RH-Firewall-1-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
 -A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT 
 -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited 
 COMMIT

Alternatively, edit the /etc/sysconfig/iptables file directly and use the command:

 service iptables restart

to have iptables read the new configuration.

NTP

• There is a known issue with guest OS clocks staying synchronized when running NTPD.
  • Red Hat Knowledge base
  • Timekeeping in VMware
    • From page 14 of this document: To enable VMware Tools time synchronization in a guest, first install VMware Tools in the guest operating system. Next, check that time synchronization is turned on. You can enable synchronization from the graphical VMware Toolbox application within the guest. Alternatively, you can set the .vmx configuration file option tools.syncTime = true to enable time synchronization. Note that time synchronization in a Linux guest works even if you are not running the VMware Toolbox application. All that is necessary is that the VMware guestd process is running in the guest and tools.syncTime is set to true.
  • In most cases NTPD should not be used. Instead, use VMware's synchronization mechanism.
    • On the HOST machine, locate the <session name>.vmx VMware session configuration file for the guest OS in question, and add a line which reads:

 tools.syncTime = "TRUE"

• edit /etc/ntp.conf and add the following lines

 server ntp.ncsa.uiuc.edu
 restrict ntp.ncsa.uiuc.edu mask 255.255.255.255 \
  nomodify notrap noquery

• edit /etc/ntp/step-tickers and add the following line

 ntp.ncsa.uiuc.edu

• If this is a Xen VM, allow NTPD to modify the clock (in DomU) by adding the following lines to /etc/sysctl.conf (in Domu):

 # Xen settings
 xen.independent_wallclock = 1

SSHD

• Edit /etc/ssh/sshd_config and set the following values

 Protocol 2
 PermitRootLogin no

• service sshd restart

Kerberos

If desired, enable kerberos authentication by following these instructions.

Yum (Fedora or Centos)

For Xen Servers

• Configure yum to set the xen kernel to be the default kernel in /etc/grub.conf anytime that yum installs a new kernel package. The default behavior is that yum will set the non-xen kernel to be the default, which will cause problems the next time the physical machine is rebooted.
  

     To change the default behavior, edit /etc/sysconfig/kernel and change the DEFAULTKERNEL to match the RPM package name of the kernel you want to be the default. For example, for a Dom0 on Fedora Core 5, this line might look like this:

   DEFAULTKERNEL=kernel-xen0
     For a DomU on FC5, the line is:
 DEFAULTKERNEL=kernel-xenU

     On Fedora Core 6, this is only a single kernel package, so the correct value is:

 DEFAULTKERNEL=kernel-xen

For All Servers

• Edit these files

 /etc/yum.repos.d/fedora-core.repo
 /etc/yum.repos.d/fedora-updates.repo
 /etc/yum.repos.d/fedora-extras.repo

and set baseurl to point to mirror.anl.gov

 baseurl=http://mirror.anl.gov/pub/fedora/linux/core/$releasever/$basearch/os/

• Apply all available updates

 #yum update

• You may receive the following:

 warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 4f2a6fd2
 Public key for gtk2-2.8.20-1.i386.rpm is not installed
 Retrieving GPG key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora
 Importing GPG key 0x4F2A6FD2 "Fedora Project <fedora@redhat.com>"
 Is this ok [y/N]:

If the "from file" is file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora, then reply yes and continue. If not, consult your local health care professional before proceeding.

• If the kernel was updated, check /etc/grub.conf
• Edit the file /etc/yum/yum-updatesd.conf to download and install updates daily:

 # how often to check for new updates (in seconds)
 # check only daily
 run_interval = 86400
 # how often to allow checking on request (in seconds)
 updaterefresh = 600
 # should we listen via dbus to give out update information/check for
 # new updates
 dbus_listener = no
 # how to send notifications (valid: dbus, email, syslog)
 emit_via = email
 # automatically install updates
 do_update = yes
 # automatically download updates
 do_download = yes
 # automatically download deps of updates
 do_download_deps = yes

• Reboot

• yum-updatesd doesn't work!!!! Switch to yum-cron (FC6 or later), as follows:

service yum-updatesd stop
rpm -e yum-updatesd
yum install yum-cron
chkconfig --level 35 yum-cron on
service yum-cron start

up2date (RHEL)

• Verify the following lines in /etc/sysconfig/rhn/up2date

 pkgSkipList=kernel*;
 ...
 pkgsToInstallNotUpdate=kernel;kernel-modules;kernel-devel;
 ...
 removeSkipList=kernel*;

• Create a daily cronjob to run the update

 sudo su -
 crontab -e

add the following line

 25 3 * * * /usr/sbin/up2date -u

Run the

 up2date -u 

command to download and install the latest versions of RHEL packages. You will have to enter the username and password of your paid Red Hat Network registration the first time you use 'up2date'.

cron-apt (Debian)

Automating Patching in debian can be done with cron-apt. (apt-get install cron-apt) There may be other ways but this is how I've done it.

• Edit /etc/apt/cron-apt/config

Change these options:

MAILTO=”someonethatwillreadit@ncsa.uiuc.edu”
MAILON=”upgrade”
OPTIONS=”-q -o Dir::Etc::SourceList=/etc/apt/security.sources.list”
cp /etc/apt/sources.list /etc/apt/security.sources.list

• Edit /etc/apt/security.sources.list

Take out everything but:

deb http://security.debian.org/ etch/updates main
deb-src http://security.debian.org/ etch/updates main

• Edit /etc/cron-apt/action.d/3-download

change

dist-upgrade -d -y

to

upgrade -u -y

Make sure it works, it should download and upgrade only security related patches and send out a mail.

Sendmail

• Verify sendmail is running and listening only on the local loopback adapter

 # lsof -i -nP|grep sendmail

 sendmail 2026     root    4u  IPv4   5308       TCP 127.0.0.1:25 (LISTEN)

The important part is the 127.0.0.1:25 in the previous output. You do not want *:25

Root Emails

• The person responsible for this machine should have a local account. That local account's home directory should have a .forward file (chmod 600) that contains their normal email address, such as xxx@ncsa.uiuc.edu.

• Edit /etc/aliases and modify the last line to point root to the user's local account mentioned in prior bullet.

 root:  user123

• Execute the newaliases command make the change effective

 newaliases

• As an alternative to using aliases, create the file /root/.forward that contains the email address of the administrator, e.g:

XXXsysadm@ncsa.uiuc.edu

Locate

Edit /etc/updatedb.conf to provide daily updates of the locate database; change DAILY_UPDATE=no to DAILY_UPDATE=yes .

Install vmware-tools

On client installations, run the vmware-tools script. It provides a faster network driver and other benefits. For some vmware server versions, the disk sharing portion of the script does not run, but that is of little consequence.

Note that you will have to run vmware-config-tools.pl again after every kernel update.

On some kernels (at least 2.6.24, perhaps others under vmware server 1.04), vmware-config-tools.pl throws off errors when attempting to create the faster vmxnet network connection. One solution requires that the vmware code on the guest be modified, which does not seem to be worth the trouble. Upgrading to vmware 1.05 might also fix the problem.

Services

• All services are turned off except for these:

 acpid          
 anacron        
 atd       
 auditd
 crond 
 dhcdbd         
 gpm            
 haldaemon      
 iptables       
 irqbalance <-- smp only AND host O/S only (not guest VMs)
 jexec      <-- unclear if really needed but "seems" like a good idea to keep
 klogd
 kudzu          
 messagebus     
 network  
 rsyslog   
 ntpd       <-- do not use if vmware guest AND running vmware-tools
 sendmail       
 smartd     <-- only on host machines (do not use on guest machines or for RAID setups)
 sshd
 syslog
 udev-post
 vmware-tools  
 xfs              <-- DO NOT START IN FEDORA 9 OR LATER (Deprecated)
 yum-cron | yum   <-- fedora installs only

Use the /usr/bin/system-config-services (X-windows GUI) or /usr/sbin/ntsysv (Text GUI). Make sure you are editing run-level 3, then click the 'Save' or 'OK' button before exiting.

• On vmware HOST machines, edit the /etc/smartd.conf file to monitor the hard drives. A typical config for 2 SATA drives is:

 /dev/sda -H -m root
 /dev/sdb -H -m root

• Reboot

Verification

• Review dmesg
• Review /var/log/messages
• Verify that you have all unnecessary listeners shut down.

 # lsof -i -nP|grep LISTEN

 sshd     1992     root    3u  IPv6   4996       TCP *:22 (LISTEN)
 sendmail 2026     root    4u  IPv4   5308       TCP 127.0.0.1:25 (LISTEN)

You should only have sshd and sendmail (on localhost only) listening for incoming connections. Of course, as you starting using this machine, you may want to open up other services, but this provides a good starting point.

DNS Registration

• Use lynx or elinks (text-based) or firefox (X-based)
• Navigate to Network Activation to register this machine with the DNS system. If you are using text based registration, it is easier to add the new host's DNS name on a graphic browser before registering.

Sun Java and GNU Java

Often, there are problems with the GNU gjc compiler and GNU JRE. While it is fairly easy to install the Sun Java binaries, it is much more difficult to get both the Sun and Gnu versions to coexist (especially if you want the Sun version to be the default). This page describes a nice way to install both. Note: uninstalling the GNU version is messy and can also cause unforeseen problems.

Jpackage.org provides a java-1.6.0-sun-compat package which sets all the symlinks in /etc/alternatives via the alternatives package. This makes it easy to switch between versions of programs (e.g. GNU Java and Sun Java or Java 5 and Java 6). However, the RPMs are tied to specific patch levels, and the packages from JPackage.org are often behind on the updates. Luckily, this page shows how you can easily create a new Sun compatibility package from the source RPM.

Note: Once you do this on one machine, you have a collection of RPMs that can be used to install Sun Java on all the machines you want. It just takes a little time for the first machine.

If you plan on installing other RPMs from JPackage.org, you may encounter a problem where /usr/bin/rebuild-security-providers cannot be found. If so, you can download and install a small RPM which will give you this executable. Assuming you have 'sudo' configured, you can run the following rpm command.

sudo rpm -Uvh http://mywiki.ncsa.uiuc.edu/mediawiki/images/8/84/Jpackage-utils-compat-0.0.1-1.fc8.noarch.rpm

Backup

If you want this machine backed up, then you will need to install the Tivoli client and contact help@ncsa.uiuc.edu to get it scheduled.

X Windows

• If your system locks up during startx, edit /etc/X11/xorg.conf and set the Driver for your video card device to vesa. The radeon driver does not seem to work on the PE1950 with ATI ES1000 RN50 cards.

 Section "Device"
         Identifier  "Videocard0"
         Driver      "vesa"
         VendorName  "Videocard vendor"
         BoardName   "ATI Technologies Inc ES1000"
 EndSection