Resolved

All back to normal again.

Apologies again for the noise and inconvenience. Words will be had.

Avatar for
Recovering

With more effort from ourselves to assist the carrier, this circuit is back up.

We are about to reintroduce it to the network.

Avatar for
Identified

This circuit's gone duff again.

We are pre-emptively shutting the core network wave on this span down to prevent any potential impact whilst this is investigated.

Avatar for
Resolved

This circuit is now back into production.

Apologies for the noise this has caused. Have a good weekend, everyone :)

Avatar for
Recovering

With a little more help for the carrier from us, the circuit has been stable in test since 16:44.

Provided that the circuit stays stable, we are going to put it back into production after 21:00.

Avatar for

We were advised by the carrier this was believed to be repaired at 06:30 (it was not). We re-attended to carry out more of our own tests and passed it back to the carrier at 11:49.

The carrier's contractors have just arrived on-site again, so hopefully we should have a resolution within the next hour or so.

Apologies for how long this is taking: even though there is no service impact (which is why I have been avoiding bombarding you with updates) it's obviously unacceptable. We were already considering alternatives for this carrier, so this just factors into that strategic planning.

Avatar for

We chased the carrier only to find out they had tested (twice, they allege) and decided the fault was not on their network, but hadn’t gotten around to telling us.

We have just attended and determined the fault is in fact on their network, and have sent the case back to the carrier for them to send someone competent out.

Apologies for the delay in clearing this case up: services remain at risk until resolved.

Avatar for
Identified

Looks like there's a fault on the underlying dark fibre circuit. We've opened a case with the carrier to investigate.

Avatar for
Investigating

We are seeing a link flapping between these sites

We are isolating the circuit until diagnostics can be carried out

Other Layer2 services should be considered at risk

Avatar for
Began at:

Affected components
  • Core Network Functions
    • Layer 2 (SOV)
    • Layer 2 (THE)