Page 1 of 1

Scheduled tor1 core network maintenance - 14-Jan-2022

Posted: Sat Jan 08, 2022 4:08 pm
by porcupine
Hi Folks,

Please be advised of the following maintenance window:

Maintenance Window: Friday January 14th, 2022 - 11:00pm - 2:00am Eastern time
Facility: Tor1
Affected Device(s): core02.tor1
Affected Customers: Tor1 general network
Service Impacting: Yes - The work in question will have the potential to cause several small outages/connectivity drops.

Details: We will be replacing the current core02.tor1 router with a spare unit we have been maintaining on-site. Unfortunately the current core02.tor1 router has begun logging memory alignment errors between the TOE (Trinity Offload Engine), and the PFE (Packet Forwarding Engine). These errors indicate a major hardware fault, and while they're not currently impacting the flow of traffic through the device, they indicate a major fault is present.

The spare router will be pre-configured with core02.tor1's exact running configuration, and put into place immediately adjacent to the current router. Sessions to the current core02.tor1 router will be gracefully shut-down, and then cables/optics moved to the new device, which will then be brought back online.

BGP customers directly connected to the core02.tor1 router will see their sessions go down/get reset during this operation as a result.

The replaced core02.tor1 router will then be returned to the original vendor, and the spare router will then be replaced with a newly procured unit to be kept as a cold-spare.

As always, we will be carefully performing the maintenance work in question, and will use every reasonable measure to attempt to mitigate any unnecessary outages/incidents/etc.

While the maintenance window in question is 3 hours long to allow staff time to prep/test/etc., we expect actual interruptions in connectivity to be limited to several substantially shorter events, much like the previous maintenance operation.