Scheduled tor1 core network maintenance - 17-Dec-2021

[porcupine]

Hi Folks,

Please be advised of the following maintenance window:

Maintenance Window: Friday December 17th, 2021 - 11:00pm - 3:00am Eastern time
Facility: Tor1
Affected Device(s): Tor1 core network
Affected Customers: General network
Service Impacting: Yes - The work in question will have the potential to cause several small outages/connectivity drops.

Details: We will be completing a number of tasks in the core network which could not be completed during previous maintenance periods due to time constraints (IE: where "We didn't get everything we wanted to do finished, but we got the things that needed to be done, finished."). This will include running several additional 10GE links between the tor1 core, and distribution layers to increase capacity, and resiliency, while moving several links between the dist01.tor1 and dist02.tor1 distribution routers to different media (to accommodate the new links to the cores).

In addition to this, we will be rolling back the OS version on the new core02.tor1 router after having discovered two bugs which had not been noticed during initial testing, but which impact the automation of systems interacting with the core network in a non-service-impacting, but distinctly problematic and undesirable manner. BGP customers directly connected to the core02.tor1 router will see their sessions go down/get reset during this operation as a result.

We expect this will be the final maintenance window to clean up the core network in the Tor1 facility, and hopefully the last maintenance window of the year.

As always, we will be carefully performing the maintenance work in question, and will use every reasonable measure to attempt to mitigate any unnecessary outages/incidents/etc., though with the extensive work being planned, we do expect to see some disruptions as a substantial portion of the core is shuffled around/new links are deployed/etc.

While the maintenance window in question is 4 hours long to allow staff time to prep/test/etc., we expect actual interruptions in connectivity to be limited to several substantially shorter events, much like the previous maintenance operation.