22-Mar-2010 - Emergency maintenance - dist01.tor1 - 10:00pm

[porcupine]

Hi Guys,

Please be advised of the following emergency maintenance window:

Maintenance Window: Monday March 22nd 2010, 10:00pm - 10:30pm EST

Affected Device: Dist01.tor1, Fast Ethernet Module #7 (Fa7)
Affected Customers: Customers physically connected to Fast Ethernet Module #7 (Fa7)
Service Impacting: Yes - Emergency hardware replacement

Details: 22-Mar-2010 - 7:56pm EST, the Fa7 module on our dist01.tor1 switch/router put all ports in an error-disabled state, due to a parity error detected within its packet buffers. We remedied this immediately, by power cycling the module in question, which resulted in affected customers traffic routing normally as of 8:08pm, after the power-on diagnostics, etc. completed.

Unfortunately, this is an indication that the module in question is no longer usable in our environment, and must be immediately replaced. We expect customers on the Fa7 module will see a 4-10 minute outage, as the module is rapidly un-cabled, replaced, and re-cabled.

Regards,

[porcupine]

Hi Guys,

This maintenance period can be considered closed. It looks like the additional downtime for customers on the Fast Ethernet #7 (Fa7) was under 3 minutes total, which we're quite pleased with:

Mar 22 22:14:06.544 EDT: %C6KPWR-SP-4-DISABLED: power to module in slot 7 set off (admin request)
Mar 22 22:14:31.006 EDT: %OIR-SP-6-REMCARD: Card removed from slot 7, interfaces disabled
Mar 22 22:16:12.025 EDT: %DIAG-SP-6-RUN_MINIMUM: Module 7: Running Minimal Diagnostics...
Mar 22 22:16:21.045 EDT: %SNMP-5-MODULETRAP: Module 7 [Up] Trap
Mar 22 22:16:20.593 EDT: %DIAG-SP-6-DIAG_OK: Module 7: Passed Online Diagnostics
Mar 22 22:16:21.613 EDT: %OIR-SP-6-INSCARD: Card inserted in slot 7, interfaces are now online