Priority Colo Inc.

Hey guys,

Well as some of you may, or may not have been aware, we have been fairly rapidly expanding, and this summer began to push the limits of what the HVAC units (Air Conditioners) could reasonably cool this summer, as temperatures started to fluctuate ever so slightly. As a result, upon signing our new contract with RACO (the entity in which we rent space from), we investigated the idea of moving to a newer suite down the hall. As a result, we've decided this is certainly for the best, and have been planning this for the past several months. Below I will outline the estimated timeframes, as well as the benefits/new features that we will all benefit from.

Benefits:

- Nearly double the space to accomodate growth
- We will be housed in with a 30 ton Liebert HVAC directly in our cage for *maximum* cooling potential (there are 2 others in the suite also)
- All cable management by RACO (with the exception of AC power) is being run from overhead trays, runners, etc. to provide more circulation under the raised flooring
- All customers are being moved from the existing 2 post racks to new 4 post cabletalk racks
- Existing cable management is being completely removed in favor of Cabletalk cable Troughs and vertical runners
- Colocation and Dedicated servers will be seperated and no longer inter-twined on APC's and racks.
- Everything in the 707 suite is new and clean, all the HVAC's are brand new, power is conditioned in the same manner as the current power (3-4 hours of UPS, dual diesel generators on the roof, redundant ATS gear, etc.)

Now to answer some basic questions (before they're asked):

- Anyone who wants to provide additional racking hardware for the 4 post racks is more then welcome to, all of which will be installed free of charge during the migration
- Racks will be 4 post, solid steel, with a rack nut and bolt/screw assembly, thus will fit any/all types of rack screws to be more accomodating (to the few people with Compaq hardware for example that does not properly screw into the racks and thus has rack shelves supporting it).
- Racks will initially be configured as 25-26" deep, which based on our measurements, will perfectly accomodate over 90% of the servers
- Everything relating to this move will be managed by our staff
- Our network will be "legged" over one segment at a time, we have cross connects to our current cage, and will simply move switches/equipment/routers/external links one at a time to minimize impact and network downtime
- The current suite (704) and the new suite (707) are less then 100 feet apart, just a walk down the hall

Impact:
- Expected downtime for dedicated servers is between 5 and 15 minutes per server, colocated servers between 5 and 20 minutes, customers who have large amounts of servers connected to a single APC/Switch 5-30 minutes (as they must all be brought down at once, then seamlessly moved to the new space), customers with external cross connects, 15-30 minutes.
- All servers will be provided with new MRTG (switch) and APC information
- Many masterswitches will be offline during the main migration, reboots may need to be handled by the pager
- Customers in Private VLAN's (that filter broadcast traffic from other areas of the other) may be temporarily moved to the default VLAN to assist in seamless migration, thus will temporarily receive more inbound "junk" broadcast traffic for a day or two.
- Migrations will be done at early morning hours, between 12:00am and 8:00am EST whereever possible
- Scheduling is for the week of 21-Mar-2004, we will be slowly moving customers over to the new location throughout the week, and will provide more personalized communication (individual email) with intended move time/dates for individual customers at this time.

If you have any questions, feel free to post them, or email us . Lets hope for a smooth migration. If theres anything I've not addressed here, questions, comments, anything, just make sure to post/email us regarding it and i'll cover it in this thread.

Regards,

In answer to a question I've got a few times:

- No, your IP addresses will not change
- No, network connectivity will not be changing.

We're simply moving our current equipment down the hall, which includes all of our uplinks to carriers, etc. Think of it as simply moving your server to a new rack, thats all it is, simply the other rack is in a different room, and all the equipment is moving there.

It looks like this move time will *not* make the week of the 21st.

RACO has *just* (this past week) been bought out by Switch And Data (www.switchanddata.com). Switch and Data is one of the largest carrier neutral colocation facilities in the world. Unfortunatly this has invalidated all of our timing. While all of the RACO staff are staying on board with the acquisition, their efforts have been concentrating on all of the work in relation to the acquisition, and have therefore fallen behind in adaquately preparing the new space.

We will post when we have another set of specific dates. I will not be sending out an email to everyone again until we can determine within reason when we will be able to move all of the equipment.

As emailed out, I thought I'd put a copy of this on the forum also for the masses, and reference.:

As noted in our original forum thread from Mid-February, we are finally getting the opportunity to relocate our equipment from the 704 suite, to the brand new 707 suite in 151 Front Street. The original thread can be found here:

viewtopic.php?t=171

Changes/Benefits:
- We will have nearly double the current space to house current equipment and future expansions, with additional adjacent space for further expansion
- Only power run under raised flooring, with all cat5 and fiber being run with appropriate ladder works, and fiber troughs, resulting in signifigantly improved airflow below the raised flooring
- All servers being moved from existing 2 post racks to rock solid 4 post racks
- Existing cable management being removed, cable trunks for patch panels run on overhead ladders, and D-rings down the back of the racks for cable management between the patch panels and the servers
- 3 new HVAC's producing signifigantly more air pressure under the raised flooring, with approximately 75 tons of HVAC capacity for a less then 5000 sqft suite (~200 BTU's of cooling per square foot)
- ** New Uplink ** - Adding Teleglobe to our BGP4 mix after the move has completed and everything has settled in
- ** New Router ** - We have acquired a new Cisco 7507 router with Dual Route Switch Process 4's (RSP4), and Dual AC Power Supplies, connectivity from NAC and Teleglobe will run through this new unit increasing capacity, and routing power to handle future expansions, and better handle Denial of Service attacks
- Custom made rack supports/wings/ears will support the end of (some) servers which do not have rail kits
- All colocation racks are 28" deep, following industry standards, Dell, Compaq, IBM, etc. rail kits should fit perfectly into the new racks, dedicated racks will remain 24" deep (the depth of the typical rackmount chassis which we build our dedicated servers into).
- KVM over IP setup is being implamented for all Dedicated Servers, and some Colocated servers, this will be available to staff only, but will result in signifigantly faster remote hands technical support.
- We will also be removing all stand alone server shelves from the facility, all colocation customers who are currently located on those shelves will be placed on rack-mountable shelves. Stand alone shelves will only be used for storage of spare parts, and as console carts.
- Colocation customers who have rackmount rail kits, or special 4 post mounting kits for their equipment (or wish to purchase them from their vendor), may supply this to us up to 1 week in advance, and we'll install it on their equipment in the new space free of charge. Rails must fit into the standard 28" deep and 19" wide 4 post racks.

Impact:
- All servers will be provided with new MRTG (switch) and APC information after the move is complete.
- APC Masterswitches may not be available during the move, same with MRTG graphs, reboots may be handled VIA the pager if you do not have functional APC information at any time during the migration.
- Customers in Private VLAN's may be temporarily moved to the default VLAN (VLAN 1) to assist in seamless migration, thus will temporarily receive more inbound "junk" broadcast traffic for the duration of the move.
- Customers will notice small cutovers as we migrate other network equipment, including our core 5513 Cisco switch, our routers, etc. Customers will not be notified for any move related downtimes that are expected to take less then 30 seconds (eg. cable swaps).
- To help ensure that the move goes smoothly, some customers may find themselves moving earlier then their expected time (in the listed overtime slots) or later then their expected move time, as needed by our staff to ensure that we do not run out of any type of resource (available switch ports, cables, apc ports, manpower, etc.) during the move.
- Customers (who host multiple servers) may be able to request to coordinate move times instead of being provided with general times, not all of these requests will be permitted for any of the various reasons outlined above.
- Since we will be utilizing 4 post racks, customers will no longer be permitted to use "mid-mount" rackmount ears, all colocation customer equipment which uses mid-mount rackmount ears has been verified that it can be mounted with the standard front/back mounts.
- Customer intervention will not be required during this move, our staff will be moving all equipment
- *IP addresses will not be changing, our network will remain the same, we are simply moving equipment down the hall to a new suite*

Having said that, we will update the afforementioned forum thread as necessary, and potentially post some pictures. Notably RACO has been bought out by Switch and Data previously this year, which is what caused the signifigant and lengthy delay in this relocation, as we can obviously only move our equipment once the new suite has been properly conditioned for us to do so.

Method:
Servers will be consoled into directly using the standard KVM console. For Linux, Windows, and FreeBSD servers, we will use the "ctrl + alt + del" keyboard combination to initiate a graceful shutdown. If the keyboard is unresponsive, or this shutdown sequence has been specifically disabled by the customer (enabled by default on all versions of FreeBSD and Linux to my knowledge), then we will attempt to SSH, or remote desktop to the server to initiate a gracefull shutdown. If this is unsucessfull, we will push the power button once. Many servers are configured to detect the power button and begin gracefull shutdown. If this is ineffective, we will hold the power button in until the server shuts down manually. Servers will then be removed from the current rack, and relocated to the new suite one at a time, with pre-run power and ethernet drops in the new suite for immediate hook up, and placement on the new racks.

Expected amounts of downtime:
Dedicated Servers - 4-10 minutes per server
Colocated Servers - 4-10 minutes per server
Webhosting/Reseller Accounts - 4-10 minutes per webhosting server. All Webhosting/Reseller servers should be relocated during the 15-Jul-2004 window.
Colocation packages with multiple servers, no switch - 4-10 minutes per server
Colocation Packages with multiple servers, and switch(es) - 2-6 minutes for each server, consequetive for all servers connected to the given switch

Move times (from starting day, to next day):
12-Jul-2004; 10:01am - 7:30am - Wiring clean-up, Customers will be removed from the core switch, and put on smaller 2900 and 3500 series switches. The 5513 switch will be removed from the current location, have both its Route Switch Module, and Redundant Supervisor engines upgraded (additional RAM, and newer releases of the IOS software), it will also have some cards replaced/upgraded. There will be several very short downtimes for each server (< 30 seconds) as wires are relocated, and customers are patched into new switches during this period. Downtime during this period should not exceed 5 minutes for any server.
13-Jul-2004; 8:00pm - 7:30am - Colocation Packages C1 - C40 will be relocated (standard window)
14-Jul-2004; 8:00pm - 7:30am - Colocation Packages C1-C40 will be relocated (late window), Colocation Packages C41 and above will be relocated (standard window), Dedicated Servers D1 - D40 will be relocated (early window)
15-Jul-2004; 8:00pm - 7:30am - Colocation Packages C41 and above will be relocated (late window), Dedicated Servers D1 - D40 will be relocated (standard window), Dedicated Servers D41 and above
will be relocated (early window)
16-Jul-2004; 8:00pm - 7:30am - Dedicated Servers D1 - D40 will be relocated (late window), Dedicated Servers D41 and above will be relocated (standard window)
17-Jul-2004; 6:00pm - 7:30am - Clean-up, anything that has not yet been moved will be relocated at this point, customers will be sent MRTG/APC information emails if possible.
18-Jul-2004; 6:00pm - 7:30am - Secondary Clean-up period, all equipment should be out of 704 at this time.

- Routers, monitoring servers, backup servers, and any miscellanious servers which are not directly providing a frontend service to any customers may be relocated at random during these periods to ensure the maximum amount of uptime, smoothness, and redundancy throughout the move.

Plans/Definitions:
Early Window - Servers will only be moved at this period if they are "stragglers" (eg. last unit on a given APC Masterswitch, thus the Masterswitch may be relocated to the new suite).
Standard Window - Time window in which you should expect your server to be moved.
Late Window - Servers will be relocated in this window if there was insufficient time in the standard window to move the server, or if there was any form of problem moving the server.

I believe this should sum everything up quite nicely for everyone, this indeed gives a detailed look at whats being done during the move (the process), and should give a default window to all customers thus they may announce maintenance as appropriate to their customers. Please remember that the scheduling window is simply a timeframe for when the given maintenance will be done, not an expected amount of downtime. If you have any questions regarding this, please do not hesitate to ask. And view the forum thread (top of email) for updates, and/or pictures as they become available.

Well guys,

So far, So good.

Monday we completed the following:

- Removal of all VLAN's, everyone put to centralized VLAN (temporarily)
- Migration of all customers from core switch sw01
- Migration of all customers to temporary Cisco 3548 and 2924 switches
- Upgrade of RSM module (on sw01), RAM and Software
- Replacement of SUP3 cards on sw01 with NFFC II SUP3 cards (with more RAM and newer software)
- Sucessfull migration of our NAC link to our new Cisco 7507 router (core1.prioritycolo.com)

Our move schedule is on time thus far, and aside from slight hinderances (eg. NAC side of link set to auto negotiate resulting in packet loss over a duplex mis-match), everything is going quite well and according to plan.

As noted, tomorrow evening (Tuesday) servers C1-C40 will be moved to the new 707 suite. Customers on said servers *may* notice a temperature drop in their servers if they monitor internal temperatures .

Let me know if you have any questions.

Regards,

Hey Guys,

Some of you may have just noticed a notice by our Nagios Network Monitor about your server being down. This was unfortunatly accurate. While moving a pair of massive 4u colocated servers, one of the cross connects was inadvertently crushed under one of the corners while it was being removed from the rack. This was not noticed immediatly, as there was a good deal of flashing on the ports still, and it did not hit any of our servers (it got about a dozen servers, a half populated Cisco 2924-xl-en switch was disconnected), thus we were not automatically paged/notified.

If this affected you and you were on the said switch, I appologise for the 10-15 minutes of downtime this caused you, along with the unnecessary nagios notifications you would have received on this matter.

I've moved the new cable since to a more out of the way spot, and this should not be an issue again (especially considering they'll all be gone by Friday/Saturday at the latest as the move is completed).

Aside from that, progress is a slight bit behind, we're sitll moving some colo's that were supposed to be moved yesterday yet didn't complete, as some servers are not shutting down, or coming back up completely cleanly (and we're committed to not moving more then two at a time to prevent us from getting a batch of downed servers causing more then a few minutes of downtime per).

Let me know if you have any questions, I hope it's being seen as smoothly from your side as it is from ours.

Regards,

With last nights window closed, I can safely say, so far so good. We are running slightly behind still on the move, thus if your server has yet to be moved, you can expect it to be done with the "late" window instead of the regular window. Several servers did not power down/up cleanly, and thus took some time to troubleshoot.

All in all though, the move is going according to plan, there has been very little issue, most units are fitting into the new racks, with our new cable management, etc. quite well.

We've also ordered some additional brackets for some of the colocated servers, specifcally the heavy ones that do not mount perfectly. They are rated for 175 pounds each, and thus will prevent there from being a build up of load on the bottom units of any given group/rack. They will be placed where needed.