Logs are deactivated while we are investigating an issue.

EDIT 19:14 UTC: Logs should now be back to normal. Sorry for the interruption.

Dedicated load balancers for Clever Cloud's own applications (APIs, Console, website, ...) are overloaded.

We are in the process of adding capacity to resolve this issue.

14:28 UTC: Performance is back to normal.

At 11:30 UTC we started getting tickets about customer's applications not responding. We started investigating. It looks like the network or the reverse proxies are responsible for that.

EDIT 12:46 UTC: we are experiencing abnormal new connection rates on public reverse proxies.

EDIT 12:50 UTC: we found the responsible application for this new connection rate and are mitigating it.

EDIT 14:19 UTC: Load balancers have been upscaled so they can handle more traffic. Performance is back to normal since 13:12 UTC.

Logs ingestion is malfunctioning. We are investigating.

08:00 UTC: New logs are being ingested. Logs emitted during the incident will not be ingested in the main logs storage system. Log drains may start receiving (part of) the older logs, we are still investigating this part.

08:15 UTC: Looks like everything that could be ingested has been ingested. Ingestion delay may still be a little higher than normal though, it should go back to normal soon.

Some instabilities have been detected on Warsaw reverse proxies, leading to some connections unexpectedly dropped. The problem has been fixed after an upgrade of said reverse proxies.

Our cellar-c1 cluster is experiencing connection issues. Some buckets might be unavailable as the network between various nodes of the cluster is currently having issues. We are investigating.

Cellar-c2 cluster isn't impacted.

EDIT 08:23 UTC: Connection seems to be back, we have notified both network providers used for Cellar-c1 and are still awaiting an answer. We are waiting a bit more to see if the links are correctly back or if we should expect another issue.

EDIT 08:47 UTC: The connection is now down again.

EDIT 09:35 UTC: The connection has been back up for 15 minutes and the root cause may have been found. We are waiting for explanations from our network provider. In the meantime, this issue may also have affected applications that are connecting to external services. We've seen loss to Scaleway and Azure, there might have been more.

EDIT 10:25 UTC: The issue now seems to be resolved. The root cause wasn't entirely found, current investigations show that a transit provider had an issue and traffic was redirected elsewhere, maybe leading to some links saturation (which would explain why the loss wasn't 100%, but more like 80%).

Deployments seem to be unresponsive at the time, we are investigating.

EDIT: The issue has been fixed

Warp 10 read operations are unavailable. We are working on it. Service should be back in ~2 hours.

Data is still being ingested.

09:45 UTC: Incident is over.

Multiple hypervisors in the Paris zone are unreachable. We are investigating.

14:52 UTC: Network issue is resolved. We are assessing the damage.

15:07 UTC: API and deployments are down. We are cleaning everything and bringing it up.

15:20 UTC: API is back. Deployments are back but have a significant delay as of now.

15:42 UTC: We are still working on this. Deployments are quicker now but not yet back to normal.

16:02 UTC: This incident is over. If you are still experiencing issues, please contact us.

Post-mortem

A maintenance operation carried out by our network provider a few hours before this incident generated a faulty BGP announce. Because of this, a significant portion of traffic coming out of our Paris infrastructure was going out via a NYC peer causing significant delay and even timeouts.

Routers in one of our Paris datacenter were heavily impacted by this issue and failed to accept configuration fixes. After multiple attempts to fix this, our provider ended up power-cycling affected routers which caused most of our hypervisors in this datacenter to be cut off from the rest of the network for 3 minutes.

Corrective actions will be taken to prevent this from happening again (BGP filters, dedicated admin network for the routers which was already scheduled to be set up in a few days). We will also make sure that we are warned in due time if a significant network configuration/hardware issue occurs.

Following https://www.postgresql.org/about/news/postgresql-133-127-1112-1017-and-9622-released-2210/, our PostgreSQL shared clusters will be upgraded to the latest minor version of their branch.

Affected clusters are:

• postgresql-c4: Paris zone
• postgresql-c5: Montreal zone

This update may affect performances of the databases and their availability.

The upgrade will start in a few minutes. This maintenance will be updated accordingly

EDIT 18:28 UTC+2: Montreal cluster is now up-to-date

EDIT 19:54 UTC+2: Paris cluster is now up-to-date but postgis extension is currently broken due to the update. We are working on a fix

EDIT 20:27 UTC+2: Paris cluster: databases are currently being migrated to a newer version of postgis. It will take a few hours to run on all of the databases

EDIT 20:42 UTC+2: This maintenance is now considered as over

An hypervisor needs to be rebooted. Customers that are impacted will shortly receive an email and add-ons that can be migrated will be migrated before the reboot. Estimated downtime is about 15 minutes.

Add-ons will start being migrated at 20:30 UTC+2. Hypervisor will be rebooted at 21:30 UTC+2

EDIT 20:36 UTC+2: Maintenance is starting. Applications are getting redeployed and add-ons are starting their migrations

EDIT 21:30 UTC+2: Add-ons that could be migrated have been migrated, applications have been redeployed. Server will now reboot

EDIT 22:00 UTC+2: Server has finished its reboot, add-ons that weren't migrated should have been reachable since 21:45 UTC+2. The maintenance is over.

A hypervisor became unresponsive in PAR zone. It's currently rebooting.

Affected applications are being automatically redeployed. Affected addons are unreachable.

21:53 UTC: The hypervisor is back online and is starting addon VMs.

21:55 UTC: All addons are back online. The incident is over.

Metrics/AccessLogs queues are being consumed. Recent data values are currently unavailable.

06:30 UTC: Incident is over.

Core services (console, API, metrics, access logs) are experiencing issues. We identified the problem and are working to resolve it.

EDIT 23:02 UTC: the incident is related to one of our hypervisors.

EDIT 23:03 UTC: we restarted the hypervisor; related databases are down.

EDIT 23:04 UTC: hypervisor is up; VMs are starting.

EDIT 23:13 UTC: metrics are down too.

EDIT 23:25 UTC: databases are up. We are now experiencing issues with our internal reverses proxies and console and API are not available.

EDIT 23:30 UTC: we queued the linked applications for a high-priority redeploy to ensure they reconnect to their databases. Core services are still partially down.

EDIT 0:00 UTC: all applications are redeployed.

EDIT 02:56 UTC: we are still working to fix issues on our internal core services (console, API); users applications/addons are not impacted.

EDIT 03:30 UTC: internal core services are back!

(All times in UTC) At 22:50 we got an alert saying access logs stopped being consumed. At 22:53 we got alerts saying hbase region servers went down.

After investigation, the hadoop namenodes were all in standby. At 23:33, after various checks, we promote one back to active. We then restarted all the hbase regionservers, then waited for the cluster to balance and heal up.

At 00:04 we restart the warp10 stores. At 00:07 everything is back to normal.

An add-on reverse proxy was restarted because of a very high load. Applications connected to that proxy may have lost connections to their add-ons. An upgrade of that proxy was planned in a few weeks to avoid any chances of high load. Other proxies were already upgraded. The upgrade will be done in the next couple of days, the proxy being now outside of the pool.

We are currently experiencing issues with the API following an update, we are rolling back to fix the issue.

16:13 - Rollback was successfully executed and everything is back to normal.

We are investigating an issue with the Paris Cellar cluster.

17:33 UTC: The issue has been resolved. It was due to a partial upgrade (in progress) of the cluster. Upgraded nodes have been downgraded.

18:08 UTC: The upgrade was in-progress to fix the security issue labelled as CVE-2021-20288. Due to the large number of machines, some of them were not yet up-to-date, which have led to the issue we were facing. Some of the machines were unable to authenticate correctly, leading to a cascading failure of multiple machines that weren't yet patched. Another strategy will be used to continue the upgrade of the cluster.

We currently have some issues with deployments. We are investigating.

Edit 22:48 UTC: The deployments should be fine since 22:30, we just made sure that everything was okay. Deployments that were stuck were restarted, those who failed can now be restarted without any issue. Sorry for any inconvenience.

Logs drains are temporarily unavailable.

EDIT 14:37 UTC - fixed.