Resolved
After 2 hours and 20 minutes

All customer facing services are believed to be fully operational at this time. If they are not, please contact the emergency support extension - 020 3026 2626 and press 9 at the menu (this option is not announced) for the fastest possible service.

There are some improvements that have been identified which would have reduced the impact to e.g. broadband customers around anycasting our DNS resolvers, additional anycasted radius instances, that we will work on in the coming days to increase resilience in the event of a total site loss in future.

We are already awaiting the required hardware for additional environmental monitoring which would allow us to more reliably determine if it is safe to start restoring services without having to be physically present on site in circumstances like this.

Please accept our apologies for the outage caused by this facility failure - we will be requesting an RFO from the facility for this incident; please email support@lchost.co.uk to request that a copy is sent to you when it is made available.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 hour and 39 minutes

We believe most services are restored but individual checks are continuing

Avatar for
Recovering
After 1 hour and 20 minutes

We have arrived on site - cooling in our area is currently operating at max output and we are working to validate the state of the SAN for the virtual platform and have re-started all dedicated servers

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 9 minutes

As we've lost resolvers in the outage in Volta, we have reconfigured our Radius to serve you Google's DNS resolvers for the time being

If you are on an affected broadband circuit, please reconnect your PPP session and your router should pick up the new resolver IPs. Alternatively, you may of course adjust your resolver configuration locally.

Avatar for
Identified

There is a cooling failure in Volta which has caused a significant amount of our equipment to go offline.

The facility are working on this and we are on our way to site.

Avatar for
Began at:

Affected components
  • Core Network Functions
    • Layer 3 (VLT)
    • Layer 2 (VLT)
  • Connectivity
    • Broadband
    • Leased Lines, EFM & EoFTTC
  • Hosting
    • Client Area & Billing System
    • cPanel Shared & Reseller Hosting
    • Dedicated Servers
    • Managed Shared Hosting
    • Virtual Server Platform (VLT)