Install missing libmhash and libmcrypt on CentOS 6 or RHEL 6

I recently began making the migration to RHEL 6. So far so good, except for a few unexpected hurdles. One of these was installing PHP on my web servers. The issue is that libmcrypt and libmhash cannot be found in the default repositories. When I attempted to build PHP, I received the following error:

configure: error: mcrypt.h not found. Please reinstall libmcrypt

Attempting to install libmcrypt-devel with only the default repositories installed will lead to a frustrating ‘nothing to do’ result from yum.

The only solution that I have found short of compiling libmcrypt from source, is to install the EPEL repositories. EPEL to the rescue!

rpm -ivh

Once installed, ‘yum install libmcrypt-devel’ will work as expected and you can continue with your configuration.

Changing the Time Zone in CentOS and RHEL

I recently checked the time on one of my servers and realized that the time was WAY off. NTP was running and forcing an NTP update with “ntpdate -u ” only corrected the time by nanoseconds. I read the date and time again, this time a little closer. It turns out that the clock was set to a European time zone for some reason.

So how do you change the time zone on a Linux server? Baffled that I had never done this before, I wanted to share with you how I did this.

All options for time zones are listed throughout /usr/share/zoneinfo. You will need to browse this folder to find the time zone appropriate for your server. For me, it was “US/Central” but I could have also chosen “America/Chicago”.

Now that we know the location of our appropriate time zone file, we need to create a symlink (or simply replace) the existing active one which is located at /etc/localtime. I prefer a symlink to prevent duplicate files whenever possible. Also, creating a symbolic link will ensure that any time zone changes (such as when Daylight Savings Time occurs – this happened to the US in 2007) will be put into place correctly. Save yourself the trouble and just create a symlink :)

In my example, I will create a symbolic link for US/Central time. Change “US/Central” to your appropriate time zone. Create a backup of localtime if you wish first…

rm /etc/localtime
ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime

Ta-da, your time zone is now changed. You can verify in a variety of ways, but the easiest would be to simply type ‘date’ at your command prompt.

Bind Query Log

Bind, bind, bind… So easy to use 95% of the time…

I recently had to debug an issue with Bind DNS queries. Thankfully this was a new instance and nothing was affected by it. One tool that proved essential for debugging this was the bind query log.

As long as you have logging configured in /var/named/chroot/etc/named.conf (assuming you chroot bind, which I highly suggest) it will work by a simple command. Here is the logging portion of my name.conf file

        channel ns01_log{
                file "/var/log/ns1.log" versions 3 size 2m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        category default{

With that configured, all you need to do to turn bind query logging on is issue this command:

[root@svr /]#rndc querylog

now that it logging has started, you can tail your ns1.log file to view the status of the incoming queries and better diagnose any queries that are causing problems.

The ‘rndc status’ command will show you whether query logging is on or not. Here is some example output:

[root@svr /]# rndc status
number of zones: 39
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/1000
tcp clients: 0/100
server is up and running

I wouldn’t recommend leaving this on for an extended period of time if you expect that the server is going to be resolving a lot of DNS queries. This file has the potential to fill up fast!

Helpful netstat commands

I’m a huge fan of netstat, however it’s arguments aren’t nearly as straight forward as many other commands. The netstat command allows you to view a variety of networking info (listening ports, active ports, active connections, etc.). The difficulty might be because the command offers so many arguments that it can be hard to find the exact ones that you want.

I’ve compiled a list below of the netstat commands that I use the most often.

Aggregate all incoming http network connections and then group them by incoming IP then sort them from least to most. This especially comes in handy for early detection of a Denial of Service attack.

netstat -napl | grep :80 | awk '{print $5}' | cut -d : -f 1 | sort -n | uniq -c | sort -n

Simply print the total number of connections coming in on port 80 (http)

netstat -an |grep :80|wc -l

Display all of the active Internet connections to this server (you may want to “|more” this command because it can get long on a busy server)

netstat -natp

Show all connections (source + destination + ports) to the server. This can also get very long!

netstat -an

Display the routing table for all ips bound to the server. This output is more than likely the exact same as the ‘route’ command

netstat -rn

Want to know what process is using which port? This command will show you which process is responsible for which port that’s actively listening or being used.

netstat -pl

Creating and Extending – Logical Volume Manager – LVM partitions

LVM has created some pretty amazing features. We can now add, remove, extend, shrink, and snapshot volumes very quickly and easily. Since the overhead of LVM is so small and there are so many features, I enjoy using LVM partitions quite a bit.

We can combine multiple drives to make them look as one. To create a new LVM partition we will create a partition with fdisk and assign it type 8e (Linux LVM), then initialize it as a physical volume, then create a volume group, then create a logical volume within that volume group, then format it and mount it. It’s that simple!
Example: Combine drives sdb and sdc into one logical volume

fdisk /dev/sdb
   #Go through fdisk process to create the partition table sdb and sdc
   #select "n   add a new partition" -> fill in to your specifications
   #select "t   change a partition's system id" -> select "8e" for Linux LVM
   #type "w" to write it out and save it.
fdisk -l                #verify partition was created
pvcreate /dev/sdb1 /dev/sdc1 #initialize both drives as physical volumes
vgcreate myVolumeGroup /dev/sdb1 /dev/sdc1 #create a volume group with these drives in it
lvcreate --name myLogicalVolume --size 100G myVolumeGroup #create a 100G partition in the volume group
mkfs.ext3 /dev/myVolumeGroup/myLogicalVolume #format it

That’s it! Now you can mount it wherever you want and begin using it. Don’t forget to add it to /etc/fstab so it shows up on your next reboot.

The process to extend a LVM partition is very similar to creating one. The best part? for 2.6 Kernels using ext3 you can do this on the fly without un-mounting your drives!
Example: add device ‘/dev/sdd’ to our newly created volume group above then extend the logical volume

fdisk /dev/sdd  
   #Go through fdisk process to create the partition table sdd
   #select "n   add a new partition" -> fill in to your specifications
   #select "t   change a partition's system id" -> select "8e" for Linux LVM
   #type "w" to write it out and save it.
fdisk -l                #verify partition was created
mkfs.ext3 /dev/sdd1
pvcreate /dev/sdd1
vgextend myVolumeGroup /dev/sdd1
lvextend -L +10G /dev/myVolumeGroup/myLogicalVolume
resize2fs /dev/myVolumeGroup/myLogicalVolume #for ext3 and 2.6 Kernels works with mounted partitions!
#Otherwise you will have to unmount the volume first

There are a few tools that you can use to verify/monitor/look at your LVM partitions. Run without any parameters will display all items that it finds.
pvdisplay – display physical volumes
lvdisplay – display logical volumes
vgdisplay – display volume groups

A little known “feature” of LVM is that by default, it reserves 5% of your blocks. This comes in handy when an LVM partition gets crippled from overflowing. You can use tune2fs to free some more space up. I wouldn’t dip into these reserved blocks if not necessary though!

tune2fs -m

Extending your swap partition in CentOS

While I’m not a fan of SWAP memory, there are certain cases that can really save you by using swap. Would you rather get out of memory errors, or have a temporary slow down?

The CentOS/RHEL manual has a great quick tutorial on how to increase your swap space on an LVM2 logical partition. Since the default swap partition on a CentOS box is /dev/VolGroup00/LogVol01, that will be used in this example. The quick 5 steps are as follows:

  1. Use swapoff to disable swapping for the associated logical swap volume
  2. lvesize the logical volume to meet your needs
  3. Use mkswap to format the new swap space so it can be used properly
  4. Use swapon to enable the newly expanded swap volume
  5. Lastly, it’s always good to verify that the swap volume has grown the proper amount

Example: Extend the current swap LVM partition

 swapoff -v /dev/VolGroup00/LogVol01
 lvm lvresize /dev/VolGroup00/LogVol01 -L +1G
 mkswap /dev/VolGroup00/LogVol01
 swapon -va
 cat /proc/swaps


Another handy trick to “create” more swap space when you aren’t using LVM or don’t have an extra partition laying around is to create a swap file yourself.

  1. Create a file (full of zeros) that is the desired size
  2. Use mkswap to format the new swap space so it can be used properly
  3. Use swapon to enable the newly created swap file

Example: Create a swap file of 1GB and add it to the swap pool

dd if=/dev/zero of=/opt/swap-file bs=1024 count=1000000
mkswap -f /opt/swap-file
swapon /opt/swap-file

The magic of the screen command

The Linux command prompt without ‘screen’ is like using a browser without tabs. I used to have 10-15 different server sessions open in different putty windows. This was a nightmare to alt+tab through to find the right server. This was also opened the door for accidentally executing a command on the wrong server. Let me welcome you to the magic of the screen command. With this command you can have limitless sessions open all in a single putty window. These sessions can have whatever you want in it and are completely separate of each other – same server, multiple servers, your game of snake, etc. The first thing to get an effective screen session setup is to setup your .screenrc file. Mine is setup with colors, 30,000 lines of scrollback, and 7 different server lables on the bottom. Note that screen starts its session numbers at 0: .screenrc:

termcapinfo xterm ti@:te@
defscrollback 30000
vbell_msg "[[[ ding ]]]"
vbell off

hardstatus alwayslastline
hardstatus string '%{= kG}[ %{G}%H %{g}][%= %{=kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{B}%Y-%m-%d %{W}%c %{g}]'

screen -t util01 0
screen -t svr01 1
screen -t svr02 2
screen -t svr03 3
screen -t svr04 4
screen -t svr05 5
screen -t svr06 6

Now that our screenrc file is set, just type the command ‘screen’ to start up a new screen session. This session will start with 7 "tabs" for each of the servers that were defined in the screenrc file. You will need to go into each "tab" and ssh to that server. Here are some helpful commands once in screen. Ctrl+a is how you start sending pretty much any command to screen. So ctrl+a then another character. This list is huge (‘man screen’ for a full list) but these are the ones that I find most helpful:

[space]     switch to the next window in line
#     switch to the specified screen number. If you have screens above #9 then you'll need to issue an apostrophe followed by the desired screen number
A     a capital A will allow you to rename the current screen's title
d     detach from the current screen session
K     Kill the current window. Very helpful if the window freezes for some reason

In addition to these, the ‘screen -ls’ bash command will show you screens that are currently attached to a socket. So if your connection to the server where you had your screen session open dies, all you need to do is reconnnect to that server and then reconnect your screen session. Your screen session does not die when you disconnect from the server. For example, I just left work and VPN’d into my network at work. I then opened a putty session to the server where my screen sessions live.

[joe@svr01 ~]$ screen -ls
There is a screen on:
        4161.pts-0.util01v      (Detached)
1 Socket in /var/run/screen/S-jchapman.

[joe@svr01 ~]$

I see that the screen session is already detached. This means that I can reattach it to my current session. If it wasn’t detached, then you would have to first detach it with -d and then reattach it to the current session with -r. So this command will reattach the screen session that is listed above:

[jchapman@util01v ~]$ screen -d 4161.pts-0.util01v

If there’s only one session listed (as there is in this example) I can also leave out the screen session name parameter and it will just connect to that one You can do all sorts of cool things like screen sharing. Hopefully this is enough for you to get up and running with screen. The man page is a great source for more information as well.  

PS1 – Changing the default bash prompt

There are all sorts of nifty tricks to customizing your bash prompt. I’m often switching between so many different servers that it its essential for me to always know the current user, server name, and current server time.

For customizing this, we have the PS1, PS2, PS3, PS4 session variables. I’m only going to touch on PS1 since that’s all I need to show me the current hostname and timestamp.

From the bash man pages:

              If set, the value is executed as a command prior to issuing each primary prompt.
       PS1    The  value  of this parameter is expanded (see PROMPTING below) and used as the primary prompt string.  The default value is
              ââ\s-\v\$ ââ.
       PS2    The value of this parameter is expanded as with PS1 and used as the secondary prompt string.  The default is ââ> ââ.
       PS3    The value of this parameter is used as the prompt for the select command (see SHELL GRAMMAR above).
       PS4    The value of this parameter is expanded as with PS1 and the value is printed before each command  bash  displays  during  an
              execution  trace.   The  first  character  of PS4 is replicated multiple times, as necessary, to indicate multiple levels of
              indirection.  The default is ââ+ ââ.

First, you can view the current variable by echoing it:

[joe@svr ~]#echo $PS1
[\u@\h \W]\$
This is the default on all of my CentOS/RHEL 5.x servers. 
The bash man pages show us a full list of options and their meanings:

       When executing interactively, bash displays the primary prompt PS1 when it is ready to read a command, and the secondary prompt PS2
       when  it  needs more input to complete a command.  Bash allows these prompt strings to be customized by inserting a number of back-
       slash-escaped special characters that are decoded as follows:
              \a     an ASCII bell character (07)
              \d     the date in "Weekday Month Date" format (e.g., "Tue May 26")
                     the format is passed to strftime(3) and the result is inserted into the prompt string; an empty format results  in  a
                     locale-specific time representation.  The braces are required
              \e     an ASCII escape character (033)
              \h     the hostname up to the first â.â
              \H     the hostname
              \j     the number of jobs currently managed by the shell
              \l     the basename of the shellâs terminal device name
              \n     newline
              \r     carriage return
              \s     the name of the shell, the basename of $0 (the portion following the final slash)
              \t     the current time in 24-hour HH:MM:SS format
              \T     the current time in 12-hour HH:MM:SS format
              \@     the current time in 12-hour am/pm format
              \A     the current time in 24-hour HH:MM format
              \u     the username of the current user
              \v     the version of bash (e.g., 2.00)
              \V     the release of bash, version + patch level (e.g., 2.00.0)
              \w     the current working directory, with $HOME abbreviated with a tilde
              \W     the basename of the current working directory, with $HOME abbreviated with a tilde
              \!     the history number of this command
              \#     the command number of this command
              \$     if the effective UID is 0, a #, otherwise a $
              \nnn   the character corresponding to the octal number nnn
              \\     a backslash
              \[     begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
              \]     end a sequence of non-printing characters

       The command number and the history number are usually different: the history number of a command is its  position  in  the  history
       list,  which  may  include commands restored from the history file (see HISTORY below), while the command number is the position in
       the sequence of commands executed during the current shell session.  After the string is decoded,  it  is  expanded  via  parameter
       expansion,  command substitution, arithmetic expansion, and quote removal, subject to the value of the promptvars shell option (see
       the description of the shopt command under SHELL BUILTIN COMMANDS below).


By looking at this, we can string together what we’re looking for:

[\t \u@\h \W]$

This will show us the current time in seconds, the username, the hostname, and then the current working directory. For example:

[16:25:36 joe@svr /]$

To test it out, you can simply set your current bash variable:

[joe@svr ~]# PS1="[\t \u@\h \W]$ "

If you like what you see, you can make it permanent by adding it to your .bashrc file (/home/user/.bashrc). This is assuming that you only want to modify your profile. otherwise you’re going to want to look in /etc/bashrc