Incident History

When you access a website or an online application, you most often do so in a “secure” way. This is for example the well-known green padlock that symbolizes HTTPS connections in your browser, which has become a standard these years thanks to initiatives like Let’s Encrypt.

This means that the data transferred to the server is encrypted, and that even if they are intercepted, they cannot be read by a third party. This protection has been provided by the TLS (Transport Layer Security) protocol for almost 20 years, whether it’s a personal site, an online shop or an access to your bank’s services.

Over time, this critical technical brick on the Internet has evolved to strengthen the level of security it offers. In August 2018, its version 1.3 (the latest) was released. Meanwhile, versions 1.0 and 1.1 were considered to no longer offer a sufficient level of protection. They have been deprecated by the IETF (Internet Engineering Task Force) since March 2021 and have therefore been gradually removed from recent browsers such as Firefox, Chrome and its derivatives or Safari.

At Clever Cloud, we have seen our customers adopt TLS 1.2 and 1.3 gradually. On our load balancers, based on our in-house and open source reverse proxy Sōzu, the latest version accounts for over 90% of the requests processed each day. TLS 1.2 for just under 9%. TLS 1.0 and 1.1 for only a few tens of thousands of requests per day, less than 0.1% of our traffic.

While we have maintained these versions for compatibility reasons, this will no longer be the case as of June 30. We will of course inform the customers affected by this choice, and encourage them to switch to more recent versions, which will have advantages for them in terms of security, performance and SEO.

Several reminders will be sent between now and the final shutdown of TLS 1.0 and 1.1. If you have any questions on this subject, please contact our support team through the Console.

EDIT 2:00 PM UTC: every public load balancers has been updated with new configuration

17:12 UTC, there was a unreported network issue. It caused two of our reverse proxies to fail. 17:13 UTC, two alerts get sent through the on-call system. The on-call person ACK both of them, handles the first one and mistake the second one for a redundant alert of the first one. 18:30 UTC, some customers complain about issues between APIs. We start investigating. 19:45 UTC, the culprit is found: a reverse proxy was down. It is restarted and everything goes back to normal. 19:50 UTC, we find the unattended alert and understand the mistake that was made. (reading the two alerts as one issue.)

A component stop consuming this queue

We are experiencing issues network connectivity issues

EDIT 14:37 UTC: Network connectivity has been resolved. Database is starting.

One of our indexes is reloading which can lead to performance issues on queries.

Due to a massive Cloudflare outage (https://www.cloudflarestatus.com) the support is not available in the ticket center. You can still contact the support via email to support@clever-cloud.com

Edit 07:13 UTC : the ticket center is back online.

One of our indexes is reloading which can lead to performance issues on queries.

EDIT 13:02 UTC: The index has reloaded

An hypervisor has been lost on the Paris zone. We are investigating.

EDIT 06:04 UTC: The server experienced a hardware failure. It may not be able to come back. Applications on it were redeployed elsewhere. Custom services and add-ons are currently impacted.

EDIT 06:23 UTC: A public reverse proxy serving requests for domain.par.clever-cloud.com (185.42.117.109) was on this hypervisor. This IP was moved to another server. Between 05:23 and 05:35, it was unreachable.

EDIT 06:52 UTC: ETA for server to come back is 08:00

EDIT 07:46 UTC: Hardware has been changed, server will be rebooted.

EDIT 07:57 UTC: Server is back online, we are making sure all services are up.

EDIT 09:10 UTC: Everything is now back to normal, the incident is over. We will investigate further on the reason of the hardware failure.

Our mail provider is currently experiencing issues. You may notice delays in receiving emails for notification, password forgotten, or account signup, billing and other services. You may also experience errors when clicking on links in those emails, like "Bad request".

EDIT 13:55 UTC: Our provider now indicates that emails should now be received with some delays.

EDIT 16:15 UTC: Email delivery should now be working fine again. Our provider's incident is over.

We are seeing an unusual amount of 503 errors on public reverse proxies, we are looking into it.

EDIT 21:28 UTC: The issue has been found and fixed. We are monitoring the situation.

EDIT 21:40 UTC: Everything seems to be back to normal. The issue was happening for a couple of applications starting around 16:30 UTC. We will investigate further on why its configuration was out of sync during that time period.

16:13:00 UTC: A hypervisor has stopped responding. We are investigating why. The system is redeploying the applications that were on it. Some reverse proxies are not responding.

16:24:00 UTC: At first look, it seems that a network error is making us see that hypervisor as down. No information yet on if it's a hardware or software network issue.

16:28:00 UTC: The hypervisor seems to be back up again. We are making sure everything on it is responding well.

16:40:00 UTC: Everything has been check and is responding correctly.

Impacts:

• Some add-ons became unresponsive.
• Logs were not served.
• One public reverse proxy was unresponsive. Traffic should have been diverted to others. Applications may have been a bit slow.
• Some custom services for customers were unresponsive.

Deployments are currently experiencing various issues, we are investigating.

EDIT 14:55 UTC: The problem has been identified and fixed. Deployments should now be working for the last 10 minutes. Sorry for the inconvenience.

An add-on reverse proxy was unreachable between 14:45 and 14:48 UTC. It has been restarted and is now serving requests as expected. Applications may have failed reaching their add-on during this time.

Our monitoring shows abnormal CPU usage on some Pulsar brokers, we are investigating.

EDIT: we stop some components which were increasing load of the cluster. it should be more stable now

[Times in UTC] 19:30: We are experiencing network issues in our Paris data center.

19:40: The culprit is a switch that half stopped responding. Turns out that it's not broken enough so its routes are automatically removed. Our DC contractor is moving to physically remove the switch. ETA is 30 minutes.

20:00: Cellar seems to be up again. We are still watching and waiting for a direct confirmation from our DC contractor.

00:00: Everything is back to normal

The unique IP service will undergo a maintenance period for 30 minutes on June 7th starting at 20:00 UTC. During this time period, the service will be unavailable. Applications using the service will encounter timeouts or various errors when trying to use the service.

Applications will automatically be restarted once the maintenance is over.

EDIT 20:05 UTC: The maintenance is beginning

EDIT 20:28 UTC: The downtime was reduced to a few minutes but multiple network cuts may have happened. Applications linked to this service are currently redeploying.

A hardware failure occurred on one of our server (hv-par4-001) Applications are being redeployed on other ones Addons are impacted

After an anormal CPU load, one of the Mongodb did not restart.

EDIT: trying to repair database files EDIT: database filesystem repaired

EDIT 04/06: MongoDB process has restarted. Some customer perform expensive queries on the MongoDB cluster, which can cause an OOM of the process,

EDIT 06/06 10:31:06 UTC: mongodb-c2 is still experiencing issues, we are working on it.

EDIT 06/06 11:24:00 UTC: Because of a replication recovery bug not fixed by MongoDB on pre-SSPL version, we are working on making databases back from the previous backups made overnight. Everything should be back on in the afternoon. Users can setup new dedicated database with the previous backups for faster recovery.

EDIT 06/06 13:45:00 UTC: Restore process has began, it will take a few hours. We will keep you posted.

EDIT 06/06 15:01:00 UTC: We restored half of the customers. We are expecting full recovery in a few hours.

EDIT 06/06 17:01:00 UTC: An issue occured while restoring the databases. We are investigating.

EDIT 06/06 23:00:00 UTC: We restored all the databases that were not above usage quota. The cluster is now running and we improved how we export connection data so applications will behave better when connecting.

Current state:

• DBs have been imported from backups. Backups that were above the free quota were not imported.
• Connection URIs have been updated to include the whole replica set. This will simplify and stabilize how applications connect to the cluster.

We identified one flaky TCP reverse proxy in the Montreal zone. We are investigating.

EDIT 20:37 UTC - fixed.

There are some issues with this service for now, applications traffic may not be routed through the proxy and may end up using another IP (hypervisor's IP) instead. We are investigating.

EDIT 09:21 UTC: The issue should have been fixed. Your applications might need to be redeployed if the issue persists. We continue to monitor the service.

EDIT 13:11 UTC: We didn't see any other issues with the service, the issue is now resolved.