Hi Everyone,
As some people will have noticed, last night at roughly 10:30pm EST, we experienced a brief hiccup affecting network connectivity.
This occurred when traffic was being redirected from our T-Systems link (due to routing issues in their core network, in an attempt to silently reroute traffic around the problem) to our other BGP links. During this operation, CPU loads on the core01.tor1 router began to spike unexpectedly as the router entered a loop while parsing the BGP table.
This resulted in our network being unavailable for roughly 7 minutes as BGP sessions on the router in question flapped several times due to the router becoming unresponsive. Following this incident, there were several more hiccups, creating 10-15 minutes of limited connectivity, as our BGP tables re converged, paths altered, etc. as the transits came back online.
I'd like to apologize for this incident, and assure customers this has been looked into in detail. Our BGP holddown timers have been adjusted (to hopefully prevent an issue like this from occurring in the future), and we will be adding an additional core router in the near future, and splitting off our peering routes to it (thus distributing the routing load a bit more, and making the configurations easier to modify, thus reducing the chances of administrative error in the future.
On an unrelated note, we will also be adding Global Crossing (GBLX) to our network shortly, which will also help to strengthen our network over the coming months.
Regards,
13-Sep-2007 - 10:30PM EST Connectivity Issues
Moderator: Admins