Incident History

An hypervisor in paris region is currently unreachable, we are investigating the issue.

Status: We are currently investigating infrastructure issues impacting multiple hypervisors in the Paris availability zone.

Update 06:22 UTC – The investigation remains in progress.

Update 06:23 UTC – The orchestration system has been temporarily stopped.

Update 06:24 UTC – A potential cooling issue has been identified in one of our Paris datacenters.

Update 06:44 UTC – Root cause investigation is ongoing, and preparations to relaunch orchestration are underway.

Update 07:08 UTC – Orchestration has been relaunched and is now catching up.

Update 07:25 UTC – We have observed that additional hypervisors in the same datacenter are also experiencing issues, and our teams are actively investigating.

Update 07:45 UTC – The datacenter team is actively addressing the cooling issue, with resolution expected by the end of the morning. In the meantime, most infrastructure has been migrated to other datacenters within the same availability zone.

A hypervisor in the PAR region has been unreachable since 08:26 UTC+2. The affected services were automatically restarted on other hosts. Our engineering team is actively investigating the root cause.

An hypervisor is unreachable on the RBX region since 11:28 UTC+2. Services on it were automatically restarted elsewhere. One of the IP behind domain.rbx.clever-cloud.com is also unreachable (87.98.177.176), it has been dropped from the DNS but you might still use it if you used A records for your domains.

We are currently investigating an issue with access logs, the system have issues processing messages.

UTC 08:36: We are currently deploying a fix

In some edge (and not common) cases, you could have some issues to reach database addons (MySQL, PostgreSQL,...). We are investigating.

EDIT:

A network configuration on MTL, MEA and GRAHDS regions was responsible for this issue. Some application instances in some very precise conditions were not able to join their databases.

A hypervisor in the RBX HDS AZ is not responding. We are investigating the issue.

Duration: 04:50 - [Current Time] UTC (Ongoing)

Affected Services: All virtual machines running on the affected hypervisor, including production workloads and their dependent services.

Impact: One hypervisor on the RBX region is experiencing erratic behavior requiring an emergency reboot. This has resulted in service interruptions for all VMs hosted on this hypervisor

Current Status: In Progress - Emergency reboot initiated. Hypervisor is currently restarting and VMs are being brought back online.

Hypervisor Issue

Issue: One of our hypervisors has crashed, causing service interruptions for addons hosted on the affected infrastructure. Applications are currently redeploying

Status: Our engineering team has been immediately notified and is actively working to restore service. We are investigating the root cause.

Duration: 14:25 - 17:45 CEST (3 hours 20 minutes)

Affected Services: Services connecting to add-ons on the Paris region through the impacted load balancer: PostgreSQL, MySQL, MongoDB, Redis, Elasticsearch, Jenkins.

Impact: One load balancer handling add-on connections on the Paris region experienced increased latency during normal operations. This may have resulted in:

• Delayed connection establishment to add-ons
• Increased data transfer times (both sending and receiving)

Current Status: Resolved - All metrics returned to normal at 17:45 CEST. The load balancer is now operating within expected parameters.

Next Steps: Root cause analysis is in progress to identify the underlying issue and prevent recurrence. We continue to monitor the situation for a few hours.

We are experiencing a communication issue between our internal rabbitmq cluster and several infrastructure components. It's blocking the deployments. We are investigating the root cause.

We are investigating issues happening with our Git repositories, some operations are failing

An hypervisor went down on PAR region We successully reboot it

Newly booted add-ons as PostgreSQL, MySQL, Redis or MongoDB are currently occuring availability issues (when using their public domain names)

We are investigating

We are investigating failures for deployments to complete. Deployments may fail without logs or any information as to why they failed. We are looking into it.

06:12 UTC : An hypervisor is down due to disk failure, we're working to bring it up again. Some specific stateful instances (databases...) can be impacted

We lost one dark fiber between two AZ of the Paris region (GDN <-> TH2) causing a redundancy loss of the connectivity between those two AZ. No customer impact is expected and we are looking into it with our providers.

We are investigating an issue with the configuration of regions and scalabilty. Users are not able to modify these parameters anymore in the information tab ("red" error message at the bottom of the console)

Edit 15:25 CEST: All users are not impacted by the issue (only Java users).

The shared cluster for our rabbitmq service is experiencing issues. A node seems to be out of the cluster while still running. We are investigating it.

An hypervisor is unreachable on Singapore, we are investigating