Resolved
After 6 days, 17 hours, and 23 minutes

All back to normal again.

Apologies again for the noise and inconvenience. Words will be had.

Avatar for
Recovering
After 6 days, 17 hours, and 2 minutes

With more effort from ourselves to assist the carrier, this circuit is back up.

We are about to reintroduce it to the network.

Avatar for
Identified
After 6 days, 3 hours, and 28 minutes

This circuit's gone duff again.

We are pre-emptively shutting the core network wave on this span down to prevent any potential impact whilst this is investigated.

Avatar for
Resolved
After 1 day, 10 hours, and 13 minutes

This circuit is now back into production.

Apologies for the noise this has caused. Have a good weekend, everyone :)

Avatar for
Recovering
After 1 day, 6 hours, and 45 minutes

With a little more help for the carrier from us, the circuit has been stable in test since 16:44.

Provided that the circuit stays stable, we are going to put it back into production after 21:00.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 day, 2 hours, and 37 minutes

We were advised by the carrier this was believed to be repaired at 06:30 (it was not). We re-attended to carry out more of our own tests and passed it back to the carrier at 11:49.

The carrier's contractors have just arrived on-site again, so hopefully we should have a resolution within the next hour or so.

Apologies for how long this is taking: even though there is no service impact (which is why I have been avoiding bombarding you with updates) it's obviously unacceptable. We were already considering alternatives for this carrier, so this just factors into that strategic planning.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 12 hours and 24 minutes

We chased the carrier only to find out they had tested (twice, they allege) and decided the fault was not on their network, but hadn’t gotten around to telling us.

We have just attended and determined the fault is in fact on their network, and have sent the case back to the carrier for them to send someone competent out.

Apologies for the delay in clearing this case up: services remain at risk until resolved.

Avatar for
Identified
After 19 minutes

Looks like there's a fault on the underlying dark fibre circuit. We've opened a case with the carrier to investigate.

Avatar for
Investigating

We are seeing a link flapping between these sites

We are isolating the circuit until diagnostics can be carried out

Other Layer2 services should be considered at risk

Avatar for
Began at:

Affected components
  • Core Network Functions
    • Layer 2 (SOV)
    • Layer 2 (THE)