j0e.us Adventures and knowledge of a Systems Engineer

Install missing libmhash and libmcrypt on CentOS 6 or RHEL 6

I recently began making the migration to RHEL 6. So far so good, except for a few unexpected hurdles. One of these was installing PHP on my web servers. The issue is that libmcrypt and libmhash cannot be found in the default repositories. When I attempted to build PHP, I received the following error:

configure: error: mcrypt.h not found. Please reinstall libmcrypt

Attempting to install libmcrypt-devel with only the default repositories installed will lead to a frustrating ‘nothing to do’ result from yum.

The only solution that I have found short of compiling libmcrypt from source, is to install the EPEL repositories. EPEL to the rescue!

rpm -ivh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Once installed, ‘yum install libmcrypt-devel’ will work as expected and you can continue with your configuration.

Changing the Time Zone in CentOS and RHEL

I recently checked the time on one of my servers and realized that the time was WAY off. NTP was running and forcing an NTP update with “ntpdate -u ” only corrected the time by nanoseconds. I read the date and time again, this time a little closer. It turns out that the clock was set to a European time zone for some reason.

So how do you change the time zone on a Linux server? Baffled that I had never done this before, I wanted to share with you how I did this.

All options for time zones are listed throughout /usr/share/zoneinfo. You will need to browse this folder to find the time zone appropriate for your server. For me, it was “US/Central” but I could have also chosen “America/Chicago”.

Now that we know the location of our appropriate time zone file, we need to create a symlink (or simply replace) the existing active one which is located at /etc/localtime. I prefer a symlink to prevent duplicate files whenever possible. Also, creating a symbolic link will ensure that any time zone changes (such as when Daylight Savings Time occurs – this happened to the US in 2007) will be put into place correctly. Save yourself the trouble and just create a symlink :)

In my example, I will create a symbolic link for US/Central time. Change “US/Central” to your appropriate time zone. Create a backup of localtime if you wish first…

rm /etc/localtime
ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime

Ta-da, your time zone is now changed. You can verify in a variety of ways, but the easiest would be to simply type ‘date’ at your command prompt.

Bind Query Log

Bind, bind, bind... So easy to use 95% of the time...

I recently had to debug an issue with Bind DNS queries. Thankfully this was a new instance and nothing was affected by it. One tool that proved essential for debugging this was the bind query log.

As long as you have logging configured in /var/named/chroot/etc/named.conf (assuming you chroot bind, which I highly suggest) it will work by a simple command. Here is the logging portion of my name.conf file

logging{
        channel ns01_log{
                file "/var/log/ns1.log" versions 3 size 2m;
                severity info;
                print-severity yes;
                print-time yes;
                print-category yes;
        };
        category default{
                ns01_log;
        };
};

With that configured, all you need to do to turn bind query logging on is issue this command:

[root@svr /]#rndc querylog

now that it logging has started, you can tail your ns1.log file to view the status of the incoming queries and better diagnose any queries that are causing problems.

The 'rndc status' command will show you whether query logging is on or not. Here is some example output:

[root@svr /]# rndc status
number of zones: 39
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is ON
recursive clients: 0/1000
tcp clients: 0/100
server is up and running

I wouldn't recommend leaving this on for an extended period of time if you expect that the server is going to be resolving a lot of DNS queries. This file has the potential to fill up fast!

Helpful netstat commands

I'm a huge fan of netstat, however it's arguments aren't nearly as straight forward as many other commands. The netstat command allows you to view a variety of networking info (listening ports, active ports, active connections, etc.). The difficulty might be because the command offers so many arguments that it can be hard to find the exact ones that you want.

I've compiled a list below of the netstat commands that I use the most often.

Aggregate all incoming http network connections and then group them by incoming IP then sort them from least to most. This especially comes in handy for early detection of a Denial of Service attack.

netstat -napl | grep :80 | awk '{print $5}' | cut -d : -f 1 | sort -n | uniq -c | sort -n

Simply print the total number of connections coming in on port 80 (http)

netstat -an |grep :80|wc -l

Display all of the active Internet connections to this server (you may want to "|more" this command because it can get long on a busy server)

netstat -natp

Show all connections (source + destination + ports) to the server. This can also get very long!

netstat -an

Display the routing table for all ips bound to the server. This output is more than likely the exact same as the 'route' command

netstat -rn

Want to know what process is using which port? This command will show you which process is responsible for which port that's actively listening or being used.

netstat -pl

Creating and Extending – Logical Volume Manager – LVM partitions

LVM has created some pretty amazing features. We can now add, remove, extend, shrink, and snapshot volumes very quickly and easily. Since the overhead of LVM is so small and there are so many features, I enjoy using LVM partitions quite a bit.

We can combine multiple drives to make them look as one. To create a new LVM partition we will create a partition with fdisk and assign it type 8e (Linux LVM), then initialize it as a physical volume, then create a volume group, then create a logical volume within that volume group, then format it and mount it. It's that simple!
Example: Combine drives sdb and sdc into one logical volume

fdisk /dev/sdb
   #Go through fdisk process to create the partition table sdb and sdc
   #select "n   add a new partition" -> fill in to your specifications
   #select "t   change a partition's system id" -> select "8e" for Linux LVM
   #type "w" to write it out and save it.
fdisk -l                #verify partition was created
pvcreate /dev/sdb1 /dev/sdc1 #initialize both drives as physical volumes
vgcreate myVolumeGroup /dev/sdb1 /dev/sdc1 #create a volume group with these drives in it
lvcreate --name myLogicalVolume --size 100G myVolumeGroup #create a 100G partition in the volume group
mkfs.ext3 /dev/myVolumeGroup/myLogicalVolume #format it

That's it! Now you can mount it wherever you want and begin using it. Don't forget to add it to /etc/fstab so it shows up on your next reboot.

The process to extend a LVM partition is very similar to creating one. The best part? for 2.6 Kernels using ext3 you can do this on the fly without un-mounting your drives!
Example: add device '/dev/sdd' to our newly created volume group above then extend the logical volume

fdisk /dev/sdd
   #Go through fdisk process to create the partition table sdd
   #select "n   add a new partition" -> fill in to your specifications
   #select "t   change a partition's system id" -> select "8e" for Linux LVM
   #type "w" to write it out and save it.
fdisk -l                #verify partition was created
mkfs.ext3 /dev/sdd1
pvcreate /dev/sdd1
vgextend myVolumeGroup /dev/sdd1
lvextend -L +10G /dev/myVolumeGroup/myLogicalVolume
resize2fs /dev/myVolumeGroup/myLogicalVolume #for ext3 and 2.6 Kernels works with mounted partitions!
#Otherwise you will have to unmount the volume first

There are a few tools that you can use to verify/monitor/look at your LVM partitions. Run without any parameters will display all items that it finds.
pvdisplay - display physical volumes
lvdisplay - display logical volumes
vgdisplay - display volume groups

A little known "feature" of LVM is that by default, it reserves 5% of your blocks. This comes in handy when an LVM partition gets crippled from overflowing. You can use tune2fs to free some more space up. I wouldn't dip into these reserved blocks if not necessary though!

tune2fs -m

Extending your swap partition in CentOS

While I'm not a fan of SWAP memory, there are certain cases that can really save you by using swap. Would you rather get out of memory errors, or have a temporary slow down?

The CentOS/RHEL manual has a great quick tutorial on how to increase your swap space on an LVM2 logical partition. Since the default swap partition on a CentOS box is /dev/VolGroup00/LogVol01, that will be used in this example. The quick 5 steps are as follows:

1. Use swapoff to disable swapping for the associated logical swap volume
2. lvesize the logical volume to meet your needs
3. Use mkswap to format the new swap space so it can be used properly
4. Use swapon to enable the newly expanded swap volume
5. Lastly, it's always good to verify that the swap volume has grown the proper amount

Example: Extend the current swap LVM partition

 swapoff -v /dev/VolGroup00/LogVol01
 lvm lvresize /dev/VolGroup00/LogVol01 -L +1G
 mkswap /dev/VolGroup00/LogVol01
 swapon -va
 cat /proc/swaps

Another handy trick to "create" more swap space when you aren't using LVM or don't have an extra partition laying around is to create a swap file yourself.

1. Create a file (full of zeros) that is the desired size
2. Use mkswap to format the new swap space so it can be used properly
3. Use swapon to enable the newly created swap file

Example: Create a swap file of 1GB and add it to the swap pool

dd if=/dev/zero of=/opt/swap-file bs=1024 count=1000000
mkswap -f /opt/swap-file
swapon /opt/swap-file