www3/www6 server replacements - 18-May-06 - 2:00am - 9:00am

[porcupine]

Hi guys,

Both the www3 and www6 servers will be migrated to the new hardware on Friday May 19th, 2006, between 2:00am, and 9:00am. We expect each server to be unavailable for 2 to 4 windows of 3-15 minutes while this migration is completed, servers are rebooted, and typical issues are quickly ironed out immediately following the migration.

Some customers will recall that the www3 server was replaced just a few months ago, unfortunately we've been experiencing problems with the RAID system, and the solution that involves the least impact to customers is a seamless migration using rsync. The www6 server has been problematic for months, primarily during backup times, while this hasn't provided a noticable disruption to customer services, it has resulted in numerous pages on a nightly basis waking our staff to see unacceptable load averages, etc.

Both servers will be upgraded to 4GB of DDR333 ECC/Registered memory, RAID1 W/hotswap drives, and CentOS 4.3. We expect the impact of these activities to be minimal, as we will be utilizing our tried and true rsync scripts, and migration methods to ensure downtime is minimal, and all applicable content has been migrated.

[porcupine]

Hi guys,

As the result of issues with the latest CPanel release and CentOS 4.3 (specifically, the MySQL install not accepting connections properly), we will be postponing this server replacement to another time.

[porcupine]

As we missed the original window for the www3 and www6 server migrations, we've been working closely with the folks at CPanel Inc., who believe they've formulated a functional solution in regards to the MySQL problems we were experiencing with this migration. As a result of this, we will be re-scheduling the migration for 26-May-2005 between 2:00am and 8:00am.

We will be upgrading the MySQL version on both production servers from 4.0.23 to 4.1.xx, and will enable the "old" MySQL 4.x style passwords prior to the migration. CPanel has informed us that doing so will allow MySQL to transfer the data correctly to the new servers, which will also be upgraded from 4.0.23 to 4.1.xx.

The specs of the new servers have not changed, nor has our estimate on the amount of time (or impact) expected by this maintenance. We expect each server to be unavailable for 2 to 4 windows of 3-15 minutes while this migration is completed, servers are rebooted, and typical issues are quickly ironed out immediately following the migration.

We have made a topic on our forums regarding this, to which you can follow for updates, etc. as applicable. Let’s hope everything tonight goes smoothly, and CPanel's suggested fix works.

[porcupine]

Both the www3 and www6 servers have been sucessfully replaced with minimal service interruption (one reboot for www6, two for www3, better then planned!).

Any customers who are experiencing any form of compatability problems, etc. should email admin@prioritycolo.com. If there are any serious issues that have become service impacting to your account, please use the emergency-pager to notify us as applicable.

[Alexander]

Just sent an emergency page to Myles, since no matter what I try e-mail is not working on several domains after the migration:

Mailwasher gives me:
Bad username or password

& ThunderBird says:
Sending of password did not success.
Mail server mail.domain.net responded:
Maildir invalid (no 'cur' directory)

& Webmail says:
ERROR: Could not complete request.
Query: SELECT "INBOX"
Reason Given: Unable to open this mailbox.

[simeon]

can't connect with pop client (Outlook) either, constant account/password popup.

[Alexander]

I hope there'll be news soon. It's only a matter of hours before ALL clients have checked e-mail and found out it doesn't work ...

[porkleton]

Dear Priority,
I'm confirming Alexander's issue with the POP mail. Please provide us with an update so that we can tell our customers something.

Thank you.

[porkleton]

Email's back up - my client is downloading all of my old messages from the server again. It appears to be downloading everything in triplicate - scratch that: quadruplicate!

[Alexander]

It was an issue with the mailbox format, quote from the e-mail Myles sent me:

... The new CPanel has no support for mbox format (only maildir it appears), and all the old servers were running mbox, so the mail formats were incompatable. While maildir is far superior to mbox, its a shame that cpanel gave no hint of those changes during install, etc. and that specific errors on the subject didn't log to /var/log/exim_mainlog

I've converted www6 to maildir format, which should boost performance a decent amount, and more importantly, get pop3 back up if I'm not mistaken...

[graham]

No Pop3 or webmail for me or my customers.

Any updates on work in progress?

[simeon]

Mails been up for us since about 8:30am this morning, but the dissappearance of neomail has got some clients upset. Some are reporting mail missing, I'm working with them to try an determine if it is truly missing or they just can't find it.

In Squirrel you can click on the Folders link and click subscribe to add in the old neomail folders.

[simeon]

.... contacts seem to be another issue though.

[gands]

Myles, what other servers have been converted to mdir?

It was an issue with the mailbox format, quote from the e-mail Myles sent me:

... The new CPanel has no support for mbox format (only maildir it appears), and all the old servers were running mbox, so the mail formats were incompatable. While maildir is far superior to mbox, its a shame that cpanel gave no hint of those changes during install, etc. and that specific errors on the subject didn't log to /var/log/exim_mainlog

I've converted www6 to maildir format, which should boost performance a decent amount, and more importantly, get pop3 back up if I'm not mistaken...

[porcupine]

Myles, what other servers have been converted to mdir?

Just www3 and www6 so far, though I'm intent on covering the rest shortly. Chances are we'll build out new hardware for the www4 and www5 servers shortly to get them into RAID1 hotswap chassis, hence they'll get converted at that time, and www2 will likely get done shortly before then.