Broadband loss of service

After a quiet 23 hours, we consider this incident to be resolved.

The root cause of this was a hardware fault within the power distribution at our wholesale supplier's datacentre facility which, while this could reasonably be expected to cause some short term disruption/reconnects, then resulted in those users who were disconnected to be unable to re-authenticate to re-connect.

They are currently investigating so as to understand what can be done, if anything, to reduce the likelihood of a similar failure occurring again, why this failure had such an extensive impact to broadband services when it ought not have, and to make whatever changes are necessary to ensure that any similar incident in future hopefully does not have such a significant impact. We will be looking forward to receiving their detailed RFO and will be working to ensure that any weaknesses uncovered are promptly addressed.

Our apologies once again for the inconvenience caused yesterday.

While we have not had any word on things being fixed, we're seeing authentication requests coming through which means that users should now be able to get online again.

If you do not currently have a connection please try reconnecting your PPP or rebooting your router.

If it still does not work, please email or slack us to let us know so that we can assess the scale of the remaining issue.

Though the lights are back on, an issue persists with the wholesale provider's authentication platform which is responsible for forwarding requests to us.

Accordingly, services that were down, remain down.

Customers who's service is currently online are encouraged not to reboot their router or otherwise cause their PPP session to reset as you may not be able to re-authenticate until the fault is cleared.

Work continues to resolve this.

Our apologies for the continuing disruption.

The wholesale provider has confirmed that one of their datacentre locations is experiencing a power fault.

Reading between the lines it seems likely that the affected lines pass through, or otherwise rely upon, this datacentre.

There is no estimated time to restore at present as the fault is still being assessed.

It appears roughly 47% of our broadband users are currently offline.

We do not yet have any further information as to the cause.

Attempts are ongoing to reach our wholesale provider to obtain some information as to the cause and likely estimated time to restore.

Apologies for the continuing inconvenience.

Our wholesale provider for ADSL/FTTC/FTTP is apparently currently experiencing near-total outage. You may find that you receive a different (likely RFC1918) IP on your connection from the one you expect, and will have no connectivity.

We are trying to get an update from them as to the cause - regrettably, it appears that their status page is also not working.

Apologies for the inconvenience - more as we have it.