Reseller/shared hosting server migrations - 18-Jun-2016

[porcupine]

Hi Folks,

Please be advised of the following maintenance window:

Maintenance Window: Saturday June 18th, 8:00pm - Sunday June 19th, 6:00am Eastern Time
Facility: Tor1 - 151 Front street west
Affected Device(s): www5.pcdc.net, www6.pcdc.net, www7.pcdc.net
Affected Customers: Shared/Reseller hosting customers
Service Impacting: Yes - Full customer migration to new hardware


Details:

We will be upgrading, and consolidating the www5.pcdc.net, www6.pcdc.net, and www7.pcdc.net servers into a single, considerably more powerful machine. The upgraded machine will include a base configuration of 40 cores (HT), 128GB of memory, and 2 RAID1 arrays (one pair of disks for general storage of the homedirs, one pair of SSD's for database/mail queues/logs/general OS/etc.). The new server will be running CentOS 7, Apache 2.4, PHP5.6, and the latest Cpanel/WHM versions, and will have plenty of room for upgrades (both in drives, RAM, etc.) if that should prove necessary.

Since the vast majority of customers (all of the reseller customers, and a number of the shared hosting customers) have dedicated IP addresses (and/or dedicated shared IP addresses), we will be moving the accounts on a per customer basis. Once a customers accounts have been moved to the new server, their IP address(es) will be removed from the old server, and added to the new server, to ensure each customer experiences the least downtime possible, without any changes necessary on the customers part. In general, little to no customer intervention should be required throughout this process, aside from updating your records to reflect the new server hostname (customers whose current/source servers are www5.pcdc.net, www6.pcdc.net, or www7.pcdc.net, your new hostname will be “www3.pcdc.net”).

Possible complications customers should be aware of/make preparations regarding:

- Customers should update their records to reflect their new server hostname as www3.pcdc.net. This is important to update in your records, since the old hostnames will cease to exist
- Customers with very old databases may find their passwords changed, in need of being reset -- Some very old databases had passwords stored in insecure formats which cannot be retrieved by MySQL v5.6, and are set to random strings (awaiting user reset) by the Cpanel account transfer tool.
- The new server will present a new security certificate (for accessing webmail/cpanel/whm/etc. by https) which is self signed and will generate an alert requiring you to view, and then accept the new certificate. This is entirely normal behaviour during a migration. Since several services utilize redirects, you may receive this message twice accessing a given service for the first time (IE: webmail)
- Customers who are using “mail5.pcdc.net”, “mail6.pcdc.net”, or “mail7.pcdc.net” in their mail settings should update these settings to reflect “mail3.pcdc.net” after the move, since the old servers/hostnames will no longer exist.
- There will be a short window where received emails may be lost (IE: received to the old server, as the customer in question is being moved to the new server/has just moved). We will shut down SMTP on the source servers for the duration of the move to help prevent this from occurring, though complications regarding this may still occur.
- Customers will want to ensure their sites/scripts/etc. are compatible with PHP 5.6, as the new servers will be running a newer PHP version (see PHP migration notes (v5.5.xx to 5.6.xx) - http://php.net/manual/en/migration56.php )
- Customers who have DNS zones for domains that are not hosted in the reseller servers whatsoever (IE: using the DNS servers to host records for entirely external domains) may need to re-create the zone files after the migration (as WHM has traditionally failed to allocate DNS zones to reseller users properly when there is no hosted account, meaning the additional DNS zones won't get copied during the migration).
- Customers existing bandwidth/stats may not migrate over cleanly, and may be lost during the upgrade (issues with some existing RRD files being corrupt/incomplete were detected with this during several of our dry-run pre-migration tests).
- Various system services may be disabled for the duration of this migration, to help protect against incidents/changes/etc. occurring during the migration itself. POP3, SMTP, IMAP, and FTP will be disabled during this window on the source servers, to prevent unexpected changes mid-migration
- The new server will be utilizing a new Cpanel configuration to send email out of each resellers IP address (instead of the main server IP address), in an effort to ensure that SPAM originating from one account does not impact other accounts. As such, SPF records will be added/updated on all accounts (to include the reseller netblock as a whole). Customers who require custom SPF records (IE: push email out through third party providers) should setup/modify the newly added SPF records to reflect this as they deem necessary (if they do not already have SPF records setup for their domain, as the entries we add will not account for third party mail providers).
- Customers using outdated themes (IE: "Monsoon", and "Bluelagoon") will find their themes have been reset to the current ("Paper Lantern") theme. The older themes have been outdated for many years, and have not been available on new Cpanel builds for several years.

All in all, we will endeavour to keep the outages on a per account basis, and as short as possible. We will be rebooting the new server several times throughout the process to effect changes, and have opened a considerable window to allow time for all individual transfers to complete, etc. -- This does not mean individual users will be offline for that amount of time, per customer outages for core services (Apache, MySQL, etc.) should be a few hits lasting several minutes each if everything goes according to plan.

Regards,