Incident History

The Pulsar product management and its dashboard are currently unavailable. The issue has been identified, and our teams are actively working on a fix.

On Saturday, January 3rd, our Paris region experienced degraded performance for approximately 6 minutes. During this period, some users encountered slower response times when accessing the platform, while others experienced intermittent network timeouts when attempting to connect to their services.

Timeline

7:37 PM UTC+1 - Our monitoring systems detected degraded performance on the Paris region. Users began experiencing increased latency and occasional connection failures.

7:43 PM UTC+1 - Service returned to normal operating conditions. All platform features became fully operational again.

Impact

During this incident, users with resources hosted in our Paris region may have noticed slower than usual response times when interacting with the platform. Some connection attempts resulted in network timeouts, requiring users to retry their requests. We can confirm that no data was lost during this incident and all services continued to run normally once connectivity was restored.

Next Steps

We are currently conducting a thorough post-mortem analysis to understand the root cause of this degradation. We will update this report with our findings.

The monitoring report two hypervisors unreachable, we are investigating.

Due to failure of an internal automated process, users may experience sampling or lag over their metrics. No data loss is expected, read path is not impacted.

Due to elevated load on one hypervisor, we are observing issues affecting applications and databases

The API is currently returning incomplete flavor information for certain runtimes. This affects the Console, the pricing page, and the CLI, which may not display all available flavors for specific endpoints.

We identified an issue affecting the access logs ingestion pipeline.

Impact:

• Access logs may be delayed or temporarily unavailable in the console
• Applications and addons continue to run normally without any impact

Due to failure of an internal automated process, users may experience sampling or lag over their metrics.

A hypervisor has crashed and remains offline. Team actively investigating and working on recovery. Addons hosted on this hypervisor are currently unavailable, apps have been redeployed.

The logs stack is currently unavailable. We are looking into it. Logs viewer from the console, clever-tools (CLI) and the API are impacted.

The central Clever Cloud API has stopped responding. It seems like the issue is in the database access.

We are investigating

An hypervisor on Paris region is unreachable, we are investigating

Public load balancers in the Paris region are experiencing increased TLS handshake times since 08:23 UTC, resulting in:

• TLS handshake timeouts for some HTTPS queries
• Extended HTTPS request processing times due to longer TLS handshake

We are investigating the root cause and mitigation options.

Incident Summary

Duration: 08:30 UTC - 10:55 UTC (2h 25m)

Impact:

Two load balancers experienced increased TLS handshake times in the Paris region, resulting in:

• TLS handshake timeouts for some HTTPS queries
• Extended HTTPS request processing times due to longer TLS handshake

Timeline:

• 08:30 UTC: First load balancer began experiencing increased TLS handshake latency
• 09:05 UTC: Second load balancer affected, amplifying customer impact
• 10:55 UTC: Incident resolved - TLS handshake times returned to normal on both load balancers

Root cause for the increased latency is under investigation. We will also review and improve monitoring thresholds and alert configurations for load balancer performance metrics to detect the issue more quickly.

An hypervisor on the SGP region is down since 15:14 UTC+1

We are working to reboot it

The cluster pulsar of Paris is currently having troubles with creation of producers, reading and consuming messages, we are investigating the underlying issue. We are investigating the issue. Impacted services are:

• Pulsar addons
• Network Groups
• Access logs
• Drains
• Logs

We are currently investigating an issue affecting the access logs ingestion pipeline.

Impact:

• Access logs may be delayed or temporarily unavailable in the console
• Applications and addons continue to run normally without any impact

An hypervisor on the PAR region has been down from 06:20 to 06:35 UTC+1

Services are now back up and we are monitoring the hypervisor

An hypervisor is unreachable on the SGP region since 14:58 UTC+2. Services on it were automatically restarted elsewhere.

Newly started Jenkins are unavailable after a deployment, we are investigating. You can reach our support team to restore the service