Tor2 UPS maintenance - 01-Aug-2024 - 9:00am - 11:00am Eastern

[porcupine]

Hi Folks,

Please be advised of the following maintenance window:

Maintenance Window: Thursday August 1st, 2024 - 9:00am - 11:00am Eastern Time
Facility: Tor2
Affected Device: UPS-A.tor2, UPS-B.tor2
Affected Customers: Tor2 Dedicated/Colocated customers
Service Impacting: No - Precautionary Notice Only

Details: During the UPS maintenance on 16-Jul-2024, the technician identified two batteries (one in UPS-A.tor2's battery cabinet, the other in UPS-B.tor2's battery cabinet) which had anomalously high resistance readings, suggesting these two batteries were degrading much more rapidly than the rest.

While the battery string is fully functional (IE: did not fail open/the load is still protected), this result indicates that the batteries would not perform properly during a power failure (IE: would not last as long as expected), which is obviously undesirable.

As a result of this, we have procured two identical replacement batteries, VIA the manufacturer's standard RMA process. We will need to put each UPS on battery bypass for approximately 10-15 minutes, while the bad batteries are replaced with new/fresh batteries. The work will be staggered, to ensure that customers who purchase primary/redundant power feeds (UPS-A & UPS-B) are not on battery bypass at the same time.

While we do not expect this to be a service impacting maintenance, each UPS unit will be operating on battery bypass while the specified batteries are being replaced, and should utility power fail at this time, there will be an interruption in services as the Automatic Transfer Switch (ATS) switches power from utility to generator. As such, this should be considered a precautionary maintenance window.


Note: we have spoken to the manufacturer regarding the failure rate of these batteries, as it's beyond what would normally be expected at this point in their life-cycle. We've reached the consensus: This is likely related to the time period in which the batteries were procured (June-2020, when many manufacturers/vendors/etc. were making various concessions, due to the pandemic, supply chain disruptions, material shortages, etc.).

Should the Tor2 UPS batteries continue to degrade at an accelerated rate, additional pro-active measures will be taken.