Incident History

Clever Cloud API behind the domain name api.clever-cloud.com got some slow-downs

We found out that there is an issue with a shard of our indexes. Some metrics may be unavailable during the reloading period.

Metrics and access logs are currently having ingestion and query issues. We are working on it.

EDIT 12:11 UTC: Metrics and logs are now accessible again. Sorry for the inconvenience.

Metrics and access logs are currently having ingestion and query issues. We are working on it.

EDIT 10:56 UTC: Metrics and logs are now accessible again. Sorry for the inconvenience.

Live logs are unavailable EDIT: an internal service was unreachable, Live Logs system is now fully operational

A mongodb node was unreachable. This node is now fixed

Metrics and access logs are currently having ingestion and query issues. We are working on it.

EDIT: Ingestion fixed, query almost restored

Logs pipeline components lost their connection EDIT: Connection issue fixed, we are consuming logs queue lag. we lost almost30 min of logs. EDIT: Lag consumed.

We are facing an issue with the indexes which result on some metrics and access logs unavailability at egress-level. EDIT: all indexes has been rebooted

Metrics and access logs are currently having ingestion issue. We are working on it.

EDIT 13:20 UTC: The issue has been fixed. Some metrics data points have been lost. Access logs are being queued for ingestion again.

Some of API calls might return a 504 error. The source cause has been found and we are working to restore the service.

EDIT 17:20 UTC: The service has been fully restored. Sorry for the inconvenience.

We have identified issues affecting logs and drains. We are working on it.

EDIT 06:45 UTC: fixed.

EDIT 07:22 UTC: we have identified another issue.

EDIT 09:45 UTC: fixed.

We are currently experiencing instabilities with our main API. We are looking into it.

EDIT 15:12 UTC: This seems to be back to normal. We did not find the root cause but we keep looking. Some actions may have failed like deployments, git push or accessing the dashboard / using the CLI in general

EDIT 17:34 UTC: We still see some instabilities, resulting in various longer queries or even errors from some services that fail to contact our API. We are still working on identifying the root cause.

EDIT 20:34 UTC: We didn't see any more instabilities since the latest status update. We'll continue to monitor the activity in the next couples of days.

Access logs currently have a few hours of ingestion delay. It is currently being resolved and the delay should be back to normal in a few hours. This impacts the retrieval of access logs using the CLI or the API. Also, the various console dashboards (status codes, requests per hours, ...) are impacted and might display out of sync data. Sorry for the inconvenience.

EDIT 20:43 UTC: The delay has now resolved, you should now be able to query the access logs using the CLI or API.

We have identified issues affecting logs and drains.

EDIT 15:05 UTC: fixed.

SYD zone is unreachable. We are investigating.

EDIT 18:23 UTC - the SYD zone (provided by OVH) seems only reachable using the OVH network

EDIT 18:30 UTC - we are waiting for our provider's feedback

EDIT 19:00 UTC - fixed https://network.status-ovhcloud.com/incidents/j5vzf90dpzcc

Metrics and access logs are currently delayed. Data points are queued and will be processed as soon as possible. This may lead to some series missing recent data.

Edit 10:27 UTC: The delay is now resolved. Sorry for the inconvenience.

An add-on reverse proxy on the PAR zone was unreachable for 15 minutes. The restart initially failed, ence the extended downtime.

This should now be resolved. The 7 other reverse proxies were working as usual.

Deployments are broken.  We are looking why.

0740: The reason has been found and it's been fixed.

We identified issues on our metrics and accesslogs storage where certain metrics and accessLogs are not accessible.

The team has found to origin. We are working on a fix.