04/07/2003 Unscheduled Downtime

Announcements concerning Networking & Related News, Planned Outages, Anything which may affect your services.

Moderator: Admins

Post Reply
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

04/07/2003 Unscheduled Downtime

Post by porcupine »

Hey Guys,

Well aren't these always the types of posts that hosts hate to make. Anyhow this is kinda post-mortem, but this is just to explain the recent ongoings (and the outage that *just* occurred). As I'm sure almost every customer noticed, we just had a downtime of just over an hour, and for the past 2 weeks, the network has been having mild to moderate hiccups as it flapped between the primary and secondary links.

The recent hiccups/flaps have been the result of the following: The main link is run down to New York, and out of Yipes (which provides a BGP4 blend of Level3, Williams, and Genuity last i checked). This link is an OC3 (155mbps), but our uplink (DCI) had hit the limits of the ATM equipment being used over the link at 112MBPS. As a result of this, any networking tech knows they cant operate a network running so right at capacity, thus they changed the interface type from a ATM interface to a POS interface. Unfortunatly as a result, the stability of the link suffered immensely.

To counteract this, DCI had arranged for a GIG-E circuit to be used between NYC and Toronto (1,000mbps) which obviously has signifigantly more capacity. Here's the rub: Unfortunatly DCI decided to do their maintenace today, without emailing us, and had merely posted status on their own page approximately 6 hours before the opening of the window. during this time they cut off their OC3 and replaced the link to their GIG-E circuit. Unfortunatly this left us with no notification, or time to notify our own clients. Hence the downtime which was just experienced.

Will this fix the last 2 weeks of instability:
- Yes, it should, the main link is no longer running off the POS OC3 interface, but only time will tell. Replacing a faulty interface should remove the issues caused by this interface.

Will we be compensated for downtime:
- Yes, if the combined downtime from all incidents this month leaves your account with more downtime then is permitted in our uptime guarantee, you will be compensated as appropriate.

Anyhow, this is being writen to prevent any confusion from repeating this in my own words a tens/hundreds of times to individiaul clients. I hope this may clear up any questions that may be in relation to "why?!" the recent problems have occurred.

Regards,
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
Post Reply