Flapping link Volta - Sovereign House

Resolved
Resolved
After 1 day, 18 hours, and 32 minutes

After >24 hours of stability, this link was re-introduced to production.

Initial investigation by the carrier suggests that the disruption was a result of their fibre crews working on a service fault for other customers in the same splice pack.

We are awaiting a full RFO from the carrier and will try to ensure that any required remedial works required to prevent a recurrence are taken under a scheduled maintenance, but this is likely to take a few weeks to progress to any kind of conclusion.

In the meantime, all services are running normally and full resilience is restored.

Apologies for any inconvenience which has been caused by this issue.

Avatar for
Identified
After 10 hours and 44 minutes

This link's flapped twice again. We're going to isolate it again out of an abundance of caution until we can get this fully inspected and tested today.

This leaves the network "at risk" of disruption in the event that there is a problem with the other side of the ring.

Avatar for
Resolved
After 4 hours and 4 minutes

We put the link back in to production at 21:53 after we were satisfied that the circuit was no longer flapping.

Apologies for the brief interruption this evening. We will be investigating this evening's issue with the carrier in question.

Avatar for
Identified

We've had some interruption to service caused by a repeatedly flapping link between Volta and Sovereign House which is about the worst possible failure mode for people who like a quiet life.

We've logged on and shut the port down whilst we investigate, which should bring all services back online, but leaves some services "at risk" until the redundant path is back online, so we'll leave this issue open until then.

Avatar for
Began at:

Affected components
  • Core Network Functions
    • Layer 2 (SOV)
    • Layer 2 (VLT)