Incident History

A hypervisor in the RBX HDS AZ is not responding. We are investigating the issue.

Duration: 04:50 - [Current Time] UTC (Ongoing)

Affected Services: All virtual machines running on the affected hypervisor, including production workloads and their dependent services.

Impact: One hypervisor on the RBX region is experiencing erratic behavior requiring an emergency reboot. This has resulted in service interruptions for all VMs hosted on this hypervisor

Current Status: In Progress - Emergency reboot initiated. Hypervisor is currently restarting and VMs are being brought back online.

Hypervisor Issue

Issue: One of our hypervisors has crashed, causing service interruptions for addons hosted on the affected infrastructure. Applications are currently redeploying

Status: Our engineering team has been immediately notified and is actively working to restore service. We are investigating the root cause.

Duration: 14:25 - 17:45 CEST (3 hours 20 minutes)

Affected Services: Services connecting to add-ons on the Paris region through the impacted load balancer: PostgreSQL, MySQL, MongoDB, Redis, Elasticsearch, Jenkins.

Impact: One load balancer handling add-on connections on the Paris region experienced increased latency during normal operations. This may have resulted in:

• Delayed connection establishment to add-ons
• Increased data transfer times (both sending and receiving)

Current Status: Resolved - All metrics returned to normal at 17:45 CEST. The load balancer is now operating within expected parameters.

Next Steps: Root cause analysis is in progress to identify the underlying issue and prevent recurrence. We continue to monitor the situation for a few hours.

We are experiencing a communication issue between our internal rabbitmq cluster and several infrastructure components. It's blocking the deployments. We are investigating the root cause.

We are investigating issues happening with our Git repositories, some operations are failing

An hypervisor went down on PAR region We successully reboot it

Newly booted add-ons as PostgreSQL, MySQL, Redis or MongoDB are currently occuring availability issues (when using their public domain names)

We are investigating

We are investigating failures for deployments to complete. Deployments may fail without logs or any information as to why they failed. We are looking into it.

06:12 UTC : An hypervisor is down due to disk failure, we're working to bring it up again. Some specific stateful instances (databases...) can be impacted

We lost one dark fiber between two AZ of the Paris region (GDN <-> TH2) causing a redundancy loss of the connectivity between those two AZ. No customer impact is expected and we are looking into it with our providers.

We are investigating an issue with the configuration of regions and scalabilty. Users are not able to modify these parameters anymore in the information tab ("red" error message at the bottom of the console)

Edit 15:25 CEST: All users are not impacted by the issue (only Java users).

The shared cluster for our rabbitmq service is experiencing issues. A node seems to be out of the cluster while still running. We are investigating it.

An hypervisor is unreachable on Singapore, we are investigating

Some deployments are currently blocked, we are investigating the issue

We are investigating various issues on the Paris region.

Metrics service is currently under heavy load and has issues to hold read/write operations. We are currently investigating and fixing the issue. Lag on metrics can be seen (few minutes at most), no data is lost.

15:14 UTC: Situation is back to normal

Following friday’s incident, FS bucket creation in several regions were prevented by a sporadic network issue.

This also prevented deployments for new PHP applications that did not disable the automatic FS bucket.

This is now resolved.

Between 12:00 UTC and 13:40 UTC all the load balancers stopped consuming orders. The monitoring alerted in a weird way, which led to a slower on-call response than expected.

The situation has been resolved

We identified availability issues on newly created add-on on the gra-hds zone. We are investigating