30-Dec-2004 - www2/www3 emergency maintenance

[porcupine]

Hi guys,

Since our mod_security, kernel upgrades, etc. we've been having a few issues. Most of them were very minor, and relating to tweaking of the mod_security rules. The www3 server has been performing sluggish, with an excessive amount of IO_WAIT load. Though we've now tried multiple kernel drivers. I've been at a loss to explain why this would be happening (it has been making the server excessively slow during backups, etc.), but I believe we may now have an explanation.

During the early releases of the 2.6.xx kernels, there were many reported problems with SCSI drives, and IO_WAIT loads. I believe these still have not been resolved for our kernel/driver/controller card/SCSI drive combinations. As such, we will be briefly upgrading the www3 server to the latest 2.4.xx kernel ( 2.4.28 ), and checking the results versus the current speed tests, and posting them here.

Customers on the www3 server, can expect between 2, and 10 minutes of downtime within the next several hours, impact should be very minimal, and complete before 9am EST (business hours for most).

Current speed:

root@www3 [~]# hdparm -tT /dev/sda

/dev/sda:
Timing buffer-cache reads: 128 MB in 0.57 seconds =222.64 MB/sec
Timing buffered disk reads: 64 MB in 27.84 seconds = 2.30 MB/sec

[porcupine]

Maintenance complete, downtime was minimal as expected (just one quick reboot).

Looks like the changes worked:

root@www3 [~]# hdparm -tT /dev/sda

/dev/sda:
Timing buffer-cache reads: 128 MB in 0.51 seconds =250.98 MB/sec
Timing buffered disk reads: 64 MB in 1.39 seconds = 46.04 MB/sec

The www3 server is now running the most recent 2.4.xx kernel. While we would definatly prefer to be running the 2.6.xx kernels, it looks like we're going to have to hold off a bit longer until we can isolate where the conflicts are coming from. The IOWait problems should now be gone, and the disk is now operating again near full speed.

[porcupine]

In light of the fact that it worked well on the www3 server, with minimal downtime, I believe that the best option now is to perform the same maintenance immediatly on the www2 server. Expected downtime is 2-4 minutes, and will be complete before 9:00am EST (business hours for most as mentioned).

Speeds before the maintenance:

root@www2 [/usr/local/src/linux-2.4.28]# hdparm -tT /dev/sda

/dev/sda:
Timing buffer-cache reads: 128 MB in 0.50 seconds =257.07 MB/sec
Timing buffered disk reads: 64 MB in 45.27 seconds = 1.41 MB/sec

[porcupine]

Maintenance complete, and again, a second sucess:

root@www2 [~]# hdparm -tT /dev/sda

/dev/sda:
Timing buffer-cache reads: 128 MB in 0.48 seconds =266.67 MB/sec
Timing buffered disk reads: 64 MB in 3.64 seconds = 17.58 MB/sec