Hi guys,
After a recent series of DoS (Denial of Service) attacks against a customer this past week, we've elected to perform the next series of router maintenance and upgrade tasks on Monday May 23rd, 2005, between 12:00am and 8:00pm. During this maintenance window, we do not expect any core services to be affected, though we will be turning down our NAC, and Teleglobe links temporarily while routers are swapped, and upgraded. Traffic will automatically be re-routing through other existing circuits (Peer1, TorIX, etc.).
We will be swapping one of our existing routers, a Cisco 7507, with the new Cisco 6k router, which should increase edge routing capacity by more then 10 times, which should be the final phase of our currently planned upgrades.
Regards,
23-May-2005 - Scheduled Maintenance - Non Service Impacting
Moderator: Admins
-
porcupine
- Site Admin
- Posts: 704
- Joined: Wed Jun 12, 2002 5:57 pm
- Location: Toronto, Ontario
- Contact:
The old core1 router has been replaced, and its relpacement router (msfc02) has been put into its old location. Customers with routes that go over our Peer1 link will have experienced several hiccups during this transition, as our BGP session with Peer1 went down several times during this maintenance.
We believe this was the result of the cross-connect fabric that provides inter-connection services for various Switch and Data customers (ourselves and Peer1 in this case). The vast majority of our cross connects are direct physical runs, though our Peer1 link goes though their switching fabric, which consists of numerous 2900, 3500, and 5500 switches. The problem we've experienced with this fabric in the past is over-subscription between the various gigabit links that connect the various Switch and Data suites (eg. in the 707 suite, there are 18/24 ports occupied on the 3550 switch, yet only one gigabit interface back to the next switch up, which means theres 1800mbps available to S&D cross connecting customers, yet only 700mbps [considering gigabit on these models is not actual gigabit] available to travel back to their other suites).
I've been talking to our Switch and Data rep. and indicating that we would like to see the various statistics (gigabit link utilization, port errors on our port, on peer1's connecting port, etc.) to determine the true nature of this incident. If the statistics indicate that this is indeed the case (and the available bandwidth on the gigabit link had been exhausted), we will move this connection away from the S&D switching fabric.
The msfc02 router is now being brought up and meshed into the network, the NAC link (only remaining link thats down, naturally traffic is re-routing around it) should be back up shortly, at which point this maintenance period will be considered closed.
We believe this was the result of the cross-connect fabric that provides inter-connection services for various Switch and Data customers (ourselves and Peer1 in this case). The vast majority of our cross connects are direct physical runs, though our Peer1 link goes though their switching fabric, which consists of numerous 2900, 3500, and 5500 switches. The problem we've experienced with this fabric in the past is over-subscription between the various gigabit links that connect the various Switch and Data suites (eg. in the 707 suite, there are 18/24 ports occupied on the 3550 switch, yet only one gigabit interface back to the next switch up, which means theres 1800mbps available to S&D cross connecting customers, yet only 700mbps [considering gigabit on these models is not actual gigabit] available to travel back to their other suites).
I've been talking to our Switch and Data rep. and indicating that we would like to see the various statistics (gigabit link utilization, port errors on our port, on peer1's connecting port, etc.) to determine the true nature of this incident. If the statistics indicate that this is indeed the case (and the available bandwidth on the gigabit link had been exhausted), we will move this connection away from the S&D switching fabric.
The msfc02 router is now being brought up and meshed into the network, the NAC link (only remaining link thats down, naturally traffic is re-routing around it) should be back up shortly, at which point this maintenance period will be considered closed.