Incident History

Logs ingestion is down. We are looking into it.

20:39 UTC: The ingestion pipeline is back for now but the underlying issue is not properly fixed yet.

20:49 UTC: Theoretically, the problem is fixed. In any case, the ingestion pipeline is working at full speed. We are keeping an eye on things.

A network incident between our two Paris datacenters occurred at 15:58 UTC and lasted for 55 seconds (with a few seconds where it was back during that time window).

We have dealt with msot consequences of that downtime, we are still working on fixing an issue with the ingestion pipeline of Metrics and access logs. There will be some delay.

16:40 UTC: Everything is working as expected, delay will go back to normal soon.

Metrics and Access logs queries are currently unavailable. Data is still ingested, only queries are impacted. ETA for resolution is 18:00 UTC.

This impacts:

• Metrics (Grafana, in the console or using or API)
• Access logs: (requests tiles for an organization / application, CLI access or our API)

EDIT 17:30 UTC: Everything is back to normal. Sorry for the inconvenience.

We are currently experiencing accessibility and delay issues on logs. We are working on it.

EDIT 21:58 UTC: Everything should be back to normal, sorry for the inconvenience.

We will migrate our support tool from Intercom to Crisp, this migration will impact your ongoing tickets with our support team, you will still be able to pursue them by replying to the e-mail transcripts but you can also change the recipient e-mail address to console+intercom@clever-cloud.on.crisp.email.

The migration will start at 19:00 UTC+1 and should apply instantly as soon as you refresh the console.

During the transition, you can directly contact us at supportmail@clever-cloud.com.

EDIT 20:44 UTC+1: The migration has ended, our new support tool is now ready to be used! Make sure to refresh the web console.

We need to do an emergency maintenance on one of our core components. This might impact deployments on all zones. Applications already deployed won't be impacted. The maintenance is starting right now.

17:26 UTC: The maintenance operation did not fix the issue. Deployments are completely disabled at the moment. We are investigating.

17:31 UTC: It was DNS (DNS reverse resolving was too slow when opening connections, which timed out). We are working on bringing everything back up.

17:52 UTC: Everything is back up. If you are experiencing an issue, please contact us.

We have experienced two network issues between the two datacenters of the PAR zone:

• Between 09:28 UTC and 09:30 UTC
• Between 09:40 UTC and 09:41 UTC

We do not have any details about this incident as of now.

You may be receiving duplicate Slack notifications.

This is due to an issue with Slack. Slack is replying with 500 errors to our notifications even though they are clearly processing the messages just fine, our notification system sends multiple retries after receving failures so you will be receiving multiple duplicates and your webhooks will probably be disabled automatically (as they are after too many repeated failures). We will be re-enabling them once the issue is fixed. If your webhook remains disabled, please contact us.

14:17 UTC: We have not received a single 500 error from Slack in 8 minutes. It looks like this may be fixed. Although a broader incident is still ongoing on Slack's end: https://status.slack.com/2021-12/a17eae991fdc437d

14:44 UTC: Webhooks disabled since 12:00 UTC have been re-enabled. Slack status says messaging/notifications part of the incident is resolved, we are not seeing any errors so this incident is now over. If you are experiencing an error or if your webhook has not been re-enabled, please contact us.

An hypervisor is unreachable in the RBX zone. Affected applications are being redeployed automatically. Affected add-ons are unreachable.

13:40 UTC: Multiple servers in the same rack have gone down at the same time. It's most likely a network issue.

13:45 UTC: Our provider (OVHcloud) is aware of the issue. They will come back to us with more details later.

13:53 UTC: The hypervisor is back online. We are making sure everything is fine.

14:11 UTC: Everything is fine now, there was an issue with outgoing traffic from 13:53 until 14:08 UTC. This is now fixed.

Our provider tells us it was an issue with the cooling system. More info may be posted here: https://bare-metal-servers.status-ovhcloud.com/incidents/5cqtb0q9ht67

We are experiencing an issue with logs ingestion pipeline. We are looking into it.

EDIT 9h15 UTC : The ingestion pipeline is back to normal. No abnormal delay.

Metrics and Access logs queries are currently unavailable. Data is still ingested, only queries are impacted. ETA for resolution is 15:00 UTC.

This impacts:

• Metrics (Grafana, in the console or using or API)
• Access logs: (requests tiles for an organization / application, CLI access or our API)

EDIT 14:52 UTC: The queries are available again since 14:20 UTC. This incident is over.

Elastic released a security bulletin regarding the impact CVE-2021-44228 has on Elasticsearch. Elastic recommends users to apply the -Dlog4j2.formatMsgNoLookups=true JVM option and restart Elasticsearch. More information in Elastic security bulletin: https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

We will apply this option on all add-ons and restart them as an emergency maintenance. For single node add-ons, this will trigger a short downtime of minimum 1 minute (the approximate time it takes Elasticsearch to boot). For clustered add-ons, no downtime is to be expected as it will be a rolling restart.

Newly created add-ons are already patched.

The restart of all add-ons will start at 15:00 UTC. Sorry for the short notice. Feel free to contact our support if you have any questions.

EDIT 15:05 UTC: Add-ons restart is starting

EDIT 16:10 UTC: Add-ons have been restarted. The maintenance is over.

An add-on reverse proxy crashed at 10:21 UTC and got restarted at 10:24 UTC. During that time, some services connecting to their add-ons might have experienced unexpected connection errors (connection lost, connection refused, ...).

The issue is now fixed.

We are experiencing an issue with logs ingestion pipeline. We are looking into it. The issue started at 23:30 UTC yesterday and was not caught until 08:02 UTC because of a missing monitoring alert following a maintenance operation a few days ago.

09:22 UTC: The issue is identified and fixed, logs ingestion should catch up. Logs should appear within a few minutes.

09:38 UTC: The issue is not actually fixed, there is something else blocking the pipeline. We are investigating.

09:55 UTC: The ingestion is working, there are a lot of older logs to be processed so it will take a while before you can see recent logs in real time.

13:07 UTC: The ingestion pipeline is back to normal. No abnormal delay.

One of our hypervisors in the Singapore zone is down.

We are investigating a network issue. We are seeing random TCP timeouts and ICMP packets dropped for a few remote hosts from some PAR hosts (very few hosts are affected by this). This started occurring on 2021-11-25 at around 22:15 UTC.

10:53 UTC: We are still investigating this issue. The culprit seems to be a peering node.

11:18 UTC: It seems to only affect a few routing paths between our infrastructure and some hosts of Scaleway and Azure. We are trying to narrow down the issue with their network teams.

13:05 UTC: We see improvements between Scaleway and our Infrastructure since 11:26 UTC. We do not yet know if it's a temporary resolution and are awaiting for more information on Scaleway side.

13:36 UTC: Confirming that the issue between Scaleway and our infrastructure has been fixed. We are still awaiting some details from Scaleway to know if they are indeed the ones who changed their routing configuration to avoid the faulty peer.

15:10 UTC: Scaleway tells us they did not change anything on their end. Still, no issue to report on this side since 11:26 UTC. On the Azure side of things, it seems to be better, the issues we could reproduce earlier cannot be reproduced anymore but some hosts may still be affected. We are marking this as resolved but if you have any specific problems, please contact us so we can troubleshoot the issue more efficiently.

One of our hypervisors needs to be shut down because of a faulty memory module. Applications have already been redeployed elsewhere and add-ons will be automatically migrated starting on December 1, 2021 at 20:30 UTC+1. Add-ons that can't be migrated will experience up to 1 hour of downtime.

Impacted users will shortly receive an email and can contact us on our technical support for any further questions.

EDIT 20:32 UTC+1: Add-ons migrations are starting

EDIT 21:31 UTC+1: Add-ons have been migrated. Add-ons that couldn't be migrated in the first place will be unavailable up to one hour. We will announce the planned downtime tomorrow (02/12/2021)

EDIT 02/12/2021: The hypervisor will be rebooted on December 06, 2021 at 11:00 UTC+1. The expected downtime is less than 1 hour.

EDIT 06/11/2021 10:59 UTC+1: The hypervisor is going down at 11:00 UTC+1 as expected. Downtime should not be higher than 1 hour.

EDIT 06/11/2021 11:09 UTC+1: The hypervisor is back up since 3 minutes, all services should be reachable again. We are making sure everything runs fine.

EDIT 06/11/2021 11:13 UTC+1: The maintenance is over.

We are experiencing an issue with logs ingestion pipeline. We are looking into it.

12:21 UTC: Incident is resolved (there may be some lag for a few minutes)

Applications using PHP 7.0 to 7.2 will be upgraded to PHP 7.4 automatically on December 1st, 2021.

PHP versions from 7.0 to 7.2 are vulnerable to security vulnerabilities as they will not receive security updates. You can find the list of end of life versions here: https://www.php.net/eol.php.

Affected customers will be e-mailed about this change and can contact our support team for any additional questions.

Access Logs and Billing are experiencing issues.

EDIT 13:05 UTC: fixed.