Incident History

We are experiencing issues with hypervisors. We are investigating.

EDIT 15:45 UTC: Two hypervisors went down. The impacted services are:

• Add-ons -> add-ons hosted on those servers are currently unavailable

• Applications -> applications that were hosted on those servers should be redeployed or in the redeploy queue

• Logs -> new logs won't be processed. This includes drains. You might only get old logs when using the CLI / Console

• Shared RabbitMQ -> A node of the cluster is down, performance might be degraded

• SSH -> No new SSH connection can be made on the applications as of now.

• FS Bucket: a FS Bucket server was on one of the servers. Those buckets are unreachable and may timeout when writing / reading files

EDIT 15:54 UTC: Servers are currently rebooting.

EDIT 15:59 UTC: Servers rebooted and the services are currently starting. We are closely monitoring the situation.

EDIT 16:07 UTC: Services are still starting and we are double checking impacted databases.

EDIT 16:11 UTC: Deployment might take a few minutes to start due to the high deployment queue.

EDIT 16:33 UTC: Most services should be back online, including applications and add-ons. The deployment queue is still processing.

EDIT 16:45 UTC: The deployment queue is now empty since a few minutes, all deployments should go through almost instantly.

EDIT 17:13 UTC: Deployment queue is back to normal.

EDIT 17:15 UTC: The incident is over.

We have detected an issue affecting our logs collection pipeline. New logs are not being ingested. We are investigating.

15:52 UTC: The issue has been identified and should be fixed. We are monitoring things closely.

16:11 UTC: Overall traffic in the logs ingestion pipeline is not completely back to normal. If one of your applications does not have up-to-date logs you can try to restart it.

16:32 UTC: We have forced the hand of a component of the ingestion pipeline making it catch up with the logs waiting in queue. It should go back to normal in a matter of minutes now.

We are investigating performance issues with the API and console. This issue seems to be caused by our dedicated reverse proxies (which do not affect the performance nor availability of our customers' applications).

While investigating the issue, something broke in one of the reverse proxies which is causing availability issues. We are working on this.

10:25 UTC: The availability issue has been resolved. We are still working on resolving the performance issue.

10:32 UTC: We found the culprit and have implemented a work-around. Performance is back to normal. We are still working on an actual fix.

Our pulsar cluster is currently having issues, we are investigating the impact it may have on the cluster's usage and how to resolve them.

EDIT 14:03 UTC: The problem is now resolved. Some connection issues happened but a retry would have worked.

16:08 UTC: The API is rejecting several deployment requests 16:10 UTC: Everything is back to normal

One of the reverse proxy was unreachable for 30 minutes on the OVH zone, this was due to an OVH networking issue. This is now fixed.

Redsmin is currently unavailable due to an expired TLS certificate. Redsmin owners have been notified, we are waiting for them to update the certificate.

EDIT 22:30 UTC: Redsmin owners updated the certificate. Redsmin should now be available again

A 15 minutes maintenance on OVH's side is planned at 06:00 UTC-5. Network might be lost during the maintenance. Only one server is going to be impacted. Applications will be redeployed. OVH status: http://travaux.ovh.net/?do=details&id=48360

EDIT 11:02 UTC: The server currently has no network. Add-ons hosted on it are currently impacted.

EDIT 11:16 UTC: The network has come back. Waiting for OVH confirmation on the end of the incident.

EDIT 11:19 UTC: OVH closed the incident, everything should be back to normal.

Metrics and access logs are currently unavailable to query. We are working on a fix.

EDIT 11:42 UTC: The issue has been fixed, metrics and access logs can be queried again. There is a delay (currently 30 minutes) in the ingestion that is currently being resolved.

EDIT 12:10 UTC: The ingestion delay is now resolved, everything should be back to normal.

Due to an issue in the advanced metrics system listing, we will need to reset it. During the next few hours, the advanced metrics will either be unavailable or display partial results for the metrics listing. Metrics will stay available, only the listing will be unavailable.

EDIT 18:53 UTC: The maintenance is still in progress.

EDIT 00:00 UTC: The maintenance is done, the custom metrics should be available again.

We are investigating an issue related to deployments. It looks like some deployments are not starting and others are not updating the reverse proxies configuration as expected.

13:54 UTC: Related to this issue, the API is unavailable at this time. We are working on it.

13:55 UTC: We stopped the deployments to avoid any more missing updates.

13:56 UTC: The API being unavailable means that the Console and the CLI will display various errors.

14:05 UTC: Git push are also unavailable, an error will occur. The main problem has been identified and we are working toward a resolution.

14:23 UTC: We are still working on fixing the root cause of this issue.

14:49 UTC: We are still working on fixing the root cause of this issue. In the mean time, we have managed to get a fully up-to-date configuration on some reverse proxies.

15:07 UTC: We believe we have fixed the root cause of the issue and are working on cleaning everything up.

15:15 UTC: Everything is looking good now. If you still have an issue, please contact us.

Today, between 17:24 UTC and 17:34 UTC, customers using our Sozu reverse proxies may have noticed errors when connecting through one of the proxies. An upgrade maintenance was ongoing which led to stop the Sozu service and a reboot of the machine. Unfortunately, the traffic wasn't correctly redirected to an alternative instance, leading to various TLS errors or HTTP errors when connecting to the non-healthy instance. Once the machine was up again, the traffic would be correctly handled.

The root cause have not yet been found but this shouldn't have happened as we routinely do such maintenance operations without any issues. We will look further into this. Apologies for the inconvenience.

Today, between 16:00 UTC and 17:50 UTC, some reverse proxies configurations updates went missing. Applications that redeployed during this time frame may have not been correctly updated on some of our reverse proxies leading to HTTP 503 / This application is redeploying or HTTP 404 / Not Found error alongside the regular applications responses.

The root cause of this is still unclear, additional investigations will be performed. A bit before 16:00, we had an incident on an internal tool that may be related.

A part of the deployment system is experiencing higher load than usual which may incur some delay before deployments actually start.

We are working on it.

16:23 UTC: This incident is over.

We are experiencing significant delay on the ingestion pipeline of Metrics.

The original incident started at around 05:15 UTC and we have been containing it since then with a lag under tens of seconds at worst.

It's now getting worse due to attempts at fixing the issue which are currently doing the opposite. This will take a while to solve.

11:17 UTC: The ingestion delay is now reduced to about 15 seconds. The issue is not completely solved, this is only a first step.

11:58 UTC: The ingestion delay is now back to normal. The root cause is not entirely fixed so this may come back but we will consider this incident as resolved for now.

Metrics in the Clever Cloud Console of applications and add-ons as well as old access logs (not the live ones) are currently unavailable. Status code charts and heat map in the application overview will also be unavailable. The system is currently recovering.

EDIT 21:10 UTC: The service is now back to normal since ~30 minutes.

A network outage is currently affecting multiple servers. We are currently investigating. Multiple services may be in degraded states or unreachable. Customers applications and databases will experience the same issues.

EDIT 17:22 UTC+1: The network have been restored on those servers. We continue investigating which services are currently impacted. Applications that lost network connectivity to our monitoring are restarting. Applications that crashed because they lost their database access are also restarting.

EDIT 17:45 UTC+1: Deployments may still take some time to start or for those ongoing, to finish. We are cleaning-up the situation.

EDIT 18:17 UTC+1: Deployments are back to normal since 18:05. We are still cleaning up the rest of the mess and making sure everything is back to normal and working fine.

EDIT 18:25 UTC+1: Incident is over.

The issue came during a maintenance of our infrastructure provider during which multiple electricity cables were disconnected on active switches. Some of our servers were linked to those switches, cutting their network access for 5 minutes. Backup network links of those servers were also affected leading to a total loss of network. We will investigate this incident further with the infrastructure provider.

The database of the core API and therefore the core API itself will be unavailable for up to 5 minutes (~ 1 minute if everything goes to plan), starting at 11:00 UTC.

11:03 UTC: The maintenance is starting, console is in maintenance mode.

11:06 UTC: Maintenance is almost over.

11:07 UTC: Maintenance is over.

We are investigating a major issue on public and internal reverse proxies (not private).

EDIT 17:55 UTC - we identified the issue (DDOS).

EDIT 17:56 UTC - we fixed the issue on internal reverse proxies.

EDIT 19:15 UTC - we are still working to fix the issue.

EDIT 20:30 UTC - fixed and situation is back to normal. We will publish a post mortem.

Post mortem 2020-11-15

16:45 UTC: Our monitoring throws an alert: public and internal reverse proxies traffic is abnormally decreasing. Dedicated reverse proxies for Premium clients are not impacted. The on-call team starts investigations;

16:53 UTC: We see a lot of HTTP requests timing out with PR_END_OF_FILE_ERROR randomly on multiple reverse proxies.

17:00 UTC: We diagnose lots of IPs running an abnormal DDoS shape on our Paris infrastructure on identified domain names which prevents reverse proxies from accepting connections and causes reduced traffic;

17:30 UTC: After banning these addresses, new ones are used for the attack and we start banning IP ranges. During this period, we are applying custom reverse proxies configurations to limit the attack impact on various clients;

17:56 UTC: We are applying these bans on the internal reverse proxies, the internal situation comes back to normal; then we ban these on public reverse proxies;

18:00 UTC: Traffic is back to normal; PR_END_OF_FILE_ERROR disappeared and we are now facing SSL_ERROR_SYSCALL. We start investigating;

18:24 UTC: We determine these errors are due to configuration errors applied during the reverse proxies configuration changes.

20:06 UTC: All configurations are fixed, everything is working as usual. We are improving reverse proxies auto-configuration to avoid error-prone manual actions. We are fixing custom clients' configuration items and are watching monitoring data closely.

20:14 UTC: Reverse proxy improved auto-configuration is deployed.

20:30 UTC: We announce the end of the incident. The attack logs will be used to improve our DDoS detection system.

For about a minute around 10:23 UTC, our servers in one of our two Paris datacenters could not reach any outside network including the other datacenter.

The impact on applications deployed on more than one scaler should be null (apart from database access depending on your particular case). Applications deployed on a single instance had about a 50% chance of being affected.

This network incident had an impact on Metrics, the service was unavailable for 15 minutes after the incident and ingestion has been delayed for another 15 minutes.

As of now, we don't know exactly what happened but we expect that a router malfunctioned and went haywire for a minute.