Failure of hypervisor
Resolved:
06:12 UTC : An hypervisor is down due to disk failure, we're working to bring it up again. Some specific stateful instances (databases...) can be impacted
Updates
07h10 UTC : we are still handling the hardware failure, we are performing the recovery process for the raid to be able to recover. The incident is impacting the development database regions, addons providers databases (which implies that the creation of addons is temporarily unavailable) and a few others.
08:21 UTC: Most addon providers services (allowing to create addons) are using their database followers. We are working to get everyone of them up and running properly. Disk recovery of the hypervisor is still running
08:21 UTC: Most addon providers services (allowing to create addons) are using their database followers. We are working to get everyone of them up and running properly. Disk recovery of the hypervisor is still running
09h00 UTC : The recovery process is still running at the raid level.
12h00 UTC : The hypervisor recovery process has failed, we are beginning the migration of every services that was running on this hypervisor.
13:30 UTC : A new fs-bucket server has been provisioned, and all apps linked to an fs-bucket will restart soon. If you have an application that failed due to an unreachable fs-bucket, you can restart it immediately to accelerate its recovery.
14h05 UTC : The hypervisor is back online and we are fixing services running on it. For services that have been restored in the meantime from a backup you will find another addon named with the suffix "[RECOVERY]".
15h30 UTC : Hypervisor is running and services are also running, the filesystem bucket hosted on it is yet available through sftp or ssh, we are working on it.