Incident History

We will start a maintenance this Sunday designed to improve performance on our storage layer for metrics and access-logs. During the maintenance, you may not see latest datapoints and access-logs.

Maintenance will start 18 of June, at 02:30 PM UTC.

EDIT 02:36 PM UTC: maintenance is starting.

Edit 08:21 PM UTC: maintenance is still on-going, storage layer is a few minutes late on average.

EDIT 08:51 PM UTC: maintenance is over, we are catching up lag

EDIT 08:00 PM UTC. An error during catching up the lag has put the storage layer into an inconsistent state. Queries are disabled for now

EDIT 11:00 PM UTC: storage layer is still inconsistent

EDIT 00:47 PM UTC D+1: storage layer is (finally?) consistent. We are catching up the lag

EDIT 04:30 PM UTC D+1: We have catch up the lag.

EDIT 07:29 AM UTC D+1: storage layer got inconsistencies. We are investigating the reason why.

EDIT 08:10 AM UTC D+1: storage layer is up and running. We are consuming the lag. Queries are disable during this phase.

EDIT 08:45 AM UTC D+1: We have consumed the lag. Queries are available.

The monitoring system has difficulties to reach some services. We are investigating...

EDIT 00:50 UTC : The monitoring do not see network issues anymore.

EDIT 01:00 UTC : The monitoring has detected connectivity issues, we are fixing.

EDIT 01:30 UTC : The monitoring has detected new connectivity issues, we are on it.

We are impacted by our infrastructure provider incident, you can get more details by following their incident website : https://network.status-ovhcloud.com/incidents/9vzvvwrm69ps

SSH connections may fail with the message 'Error: This application has no instances you can ssh to' or may ask you a password during the connection initialization. We are currently investigating this issue.

08:10 UTC : We have found the component causing this issue and restarted it. We are still investigating the root cause.

21/06 : The problem was most likely caused by the network instability observed at this time. We haven't detected any problems since.

One hypervisor only responds to ping. It does not take new VMs anymore and does not delete VMs that should be deleted.

19:57 UTC: We are going to reboot it. Some databases (that run on this hypervisor) will become unresponsive for a few minutes.

20:18 UTC: Hypervisor has been rebooted. All services hosted on it have been checked: everything is up and running.

Logs show a kernel panic.

Live logs system storage layer falls in read-only mode. we are investigating the issue.

EDIT 09:30 UTC : Following the incident https://www.clevercloudstatus.com/incident/669, the storage layer did not perform scheduled tasks.

EDIT 09:45 UTC : The storage layer is accepting write. Logging system is operating normally.

We are investigating a network connectivity issue towards our Paris region.

EDIT 00:27 UTC: The issue has been identified and fixed around 00:11 UTC. We continue identifying the impact on customer and internal services.

EDIT 01:00 UTC: We have identified services impacted by the incident and we have started to recover from the network issue. Identified impacted services are Metrics and access logs that are taking time to recover, others services should be working normally.

EDIT 02:30 UTC: Metrics and access logs are recovering from the network issue.

EDIT 04:00 UTC: Metrics and access logs are still recovering from the network issue. To follow, the incident you can go on https://www.clevercloudstatus.com/incident/669

Following the incident https://www.clevercloudstatus.com/incident/669, we are recovering the network connectivity issue

EDIT 06:05 UTC: The storage layer is now up and healthy. We are now consuming the ingestion lag, it should take a few hours to fully resolve. Queries are now available but will show outdated data. We will update this status accordingly.

EDIT 10:00 UTC: We've had a slower ingestion than initially anticipated so queries are still returning out of date data. We've made some adjustments and saw an increase in ingestion for the last hour. We will still need a few hours to fully consume the lag.

EDIT 15:00 UTC: The lag has been consumed, the metrics and access logs stack is operating normally.

The monitoring system has detected that an hypervisor is unreachable. We are investigating.

EDIT 08:32 UTC : We have found the issue and the hypervisor is rebooting

EDIT 08:50 UTC: The hypervisor has finished to reboot and services is working

An hypervisor rebooted on the Paris zone. Impacted applications are redeployed on other servers. We are monitoring the situation.

EDIT 11:40 UTC: All impacted applications have been redeployed automatically. We will investigate further why this server rebooted. The incident is now over.

Our metrics system's hbase cluster is in an inconsistent state. We found out which nodes are responsible for it and are fixing them.

12:26 UTC: we restarted the node responsible for the issue. While it re-converges, we stop the egress servers. We will put them back on in a few minutes.

13:31 UTC: Query is back online. We are still catching up the lag, so new datapoints may not be available

14:35 UTC: lag has ben catched up

Our monitoring has detected failure on the storage layer of metrics and access logs. We have found that a storage node has lost several disk. We have remove faulty disks and restarted the storage node.

EDIT 16:00 UTC : The storage layer is restarted and we are consuming the ingestion lag

• 2023-06-07 08:56 UTC: A hypervisor on the RBX zone has rebooted.
• 09:00: the machine has fully rebooted, it is restarting all its VMs. Applications VMs are redeploying on other hypervisors.
• 09:31: the checks are done, everything seems to be running fine as of now.

We will investigate to understand why this hypervisor rebooted in the first place.

Monitoring of load balancers is detecting an abnormal amount of http 404 status. We are investigating.

EDIT 13:00 UTC : We have located the root cause, we are applying a fix.

EDIT 14:20 UTC : The issue is resolved

We lost connectivity with an hypervisor on RBX. Applications have been redeployed but some databases may not be reachable. We are investigating.

EDIT 03:58 UTC: server is back online. All databases should now be reachable.

We are detecting some errors on our storage layer responsible for storing metrics and access logs data. We are investigating.

EDIT Lag has been catched up

2023-06-01 16:20 UTC : During the RBXHDS incident, one of the Cellar LB lost its configuration. The configuration of each LB was not correctly monitored. Only the whole service availability was.

2023-06-02 09:15 UTC : after customer complaints we found out about the LB misconfiguration and fixed it.

2023-06-02 09:28 UTC : monitoring checks have been added to catch this kind of issues right away.

Monitoring of load balancers is detecting an abnormal amount of http 404 status. We are investigating.

EDIT 17:51 UTC : We have found the issue and the fix is passed. Everything is operating normally

We are currently aware of an issue impacting our Ticket center service. This may impact our customers to open, view and reply to the tickets opened with our support team.

EDIT 13:30 UTC: Our ticket center provider told us that the issue has been mitigated on their end and that it is now resolved. We keep monitoring the situation for now but we can indeed see that service are operating normally those last few minutes.

EDIT 14:47 UTC: We did not see any other issues. We consider this incident to be over.

The monitoring detect errors on the metrics / access logs storage layer. We are investigating.

EDIT 11:46 UTC : We have found the issue and fixed it. We are recovering the lag.

EDIT 13:19 UTC: The lag has been consumed, everyhting is operating normaly