Incidents

Our Paris region had availability issues from certain networks. The issue was coming from one of our network transit provider and started around 22:40 UTC+2. We stopped the peering session with our transit provider at 23:50 UTC+2 and the situation is now back to normal.

Traffic incoming or outgoing through this transit provider might have been lost or had increased latencies during this incident.

We continue to monitor the situation.

08:19 UTC: Our ingresses & egresses received no more requests from customers

08:25 UTC: We re-established the connectivity with sampling on data. 08:35 UTC: Incident solved

We are investigating an issue which prevents from creating new accounts or organisations.

A hypervisor crashed in our PAR region. It rebooted itself 3 minutes later.

We are currently checking and restarting all the services it holds.

We are currently experiencing an unexpected reboot of a hypervisor in our Paris region data center. This incident has led to temporary disruptions in service for some of our hosted services on that hypervisor. We are investigating the cause of the reboot and working to restore normal operations as quickly as possible. We are prioritizing the recovery of critical services and applications.

At 15:15 UTC a database upgrade made a bug in the API visible. Applications could not be created when following the console’s application creation screen.

The bug was found and fixed.

We are investigating an outage impacting service availability for heptapod.host.

The deployment API is currently unavailable, leading to crashed applications to remain inaccessible until the issue is resolved. However, the Clever Cloud API is functioning properly, while the web console is currently down. On-call teams are diligently working on resolving the issue.

EDIT 16:12 UTC: We are restarting the deployment stack.

EDIT 16:48 UTC: The deployments have resumed, and the team is closely monitoring this system in particular.

EDIT 16:59 UTC: All systems appear to be functioning properly. The team will continue monitoring the situation.

EDIT 17:48 UTC: All systems continue to function properly. Only some instances initialization on one of our PAR AZ is still triggering occasional errors. We continue to investigate it.

A hypervisor has crashed and is currently rebooting. It has an impact on deployments in par6 availability zone. All the databases located on this machine are currently down.

The SSH gateway fails to setup the temporary keys on the VMs. This comes from a certificate issue with the underlying AMQP cluster.

We are investigaging the source cause.

We are monitoring our mailing system for our status page. Some emails seem not to be properly delivered

The GRAHDS region is currently unreachable. This seems to be a problem in OVHcloud network as we can't reach it from the public internet or from other OVHcloud regions.

We are investigating an issue with emailing of Clever Cloud Status.

An hypervisor on the PAR region is unreachable and is currently rebooting. Services hosted on this hypervisor are also unreachable.

An hypervisor on the PAR region was unresponsive between 11:48 CEST and 11:54 CEST. Applications on it are queued for redeployment. During this time, services on the hypervisor may have failed to respond or have elevated response time.

EDIT 12:53 CEST: All applications were redeployed before 12:12 CEST. The incident is now over.

[16:12] (CET) After migrating all remaining addon on MySQL 8.0 MTL DEV cluster, we will shutdown it

We had an issue on the ingestion pipeline of access logs, we are recovering from the issue and consuming the lag. The lag will be fully consume by the end of the day.

We will migrate soon www.clevercloudstatus.com to a newer version of the underlaying software for a better experience !

No disruption should occur.

Please notify the support if you notice any issue.

[15h26] (CET) After migrating all remaining addon on PostgreSQL 11 MTL DEV cluster, we will shutdown it

[13:30 CET] - Grafana instances failed after a maintenance restart

[14:50 CET] - Grafana is accessible, but alerting is deactivated: grafana alerting system failed restarting. We are currently investigating

[18:22 CET] - Grafana is accessible and alerting is reactivated: our proxy cause our database connection to failed during migration causing connection latency and multiple errors that prevent the Grafana to start.

[20:15 CET] - Grafana is accessible now in 9.5.13 - End of incident