Resolved
Resolved
After 1 hour and 43 minutes

Everything's still working. We'll be back another time to swap out this PDU, just in case.

Apologies for the interruption to your evening.

Avatar for
Recovering
After 1 hour and 19 minutes

We have resolved the power issue in THN, and directly connected services should be recovered, or recovering.

The root cause appears to have been related to the PDU itself, but whatever happened, it wasn't enough to trip the incoming supply. Resetting the main switch on the PDU was sufficient to restore power. There are no failed components upstream of the rack PDU that could explain this behaviour.

On that basis, I'll stay on site for the next 30 minutes or so to monitor, and if no problem occurs we will schedule a maintenance in the near future to replace this PDU rather than doing it unplanned.

Avatar for
Investigating
After 1 hour and 27 seconds

There’s a power issue in the rack. I’m working on identifying the actual underlying cause.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 27 minutes

We believe that broadband services should now be fully functional.

The router in Telehouse North is still AWOL, so.. I'm on my way.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 7 minutes

We’ve lost a core router in Tehouse North. This will be impacting services delivered directly in Telehouse North and is currently preventing one of our lns from operating normally causing the problems for broadband customers.

We’ll implement a workaround for the lns first and then head into Telehouse North if necessary. Please bear with us!

Avatar for
Identified
After 5 minutes

We've confirmed there is a problem, we're working to resolve it.

Avatar for
Investigating

We are investigating reports of problems with broadband services, but initial indications suggest a wider network incident. More as we have it.

Avatar for
Began at:

Affected components
  • Core Network Functions
    • Layer 3 (THN)
    • Layer 2 (THN)
  • Connectivity
    • Backhaul Services
    • Broadband
    • Leased Lines, EFM & EoFTTC