Incidents

We identified issues on our metrics and accesslogs storage where certain metrics and accessLogs are not accessible.

Problem has been identified, we are working to fix the problem.

EDIT 15:36 UTC: certain metrics and accessLogs are still not accessible. EDIT 18:50 UTC: metrics and accessLogs are now accessible.

Some parts of our infrastructure are slowing down the deployments.

Our private reverse proxies (which serve our APIs) are encountering performance issues. This is slowing down API requests and parts of the deployment process.

We are trying to fix these performance issues.

We have identified issues affecting logs and drains.

EDIT 18:43 UTC: fixed.

Due to security issues in the biscuit-auth token v1. The addon pulsar cluster will be restarted with the new biscuit authentication/authorization plugins (biscuit v2.0) which have breaking changes. The related addons will have their environment variables updated accordingly so the linked applications will be redeployed automatically.

Everything went well. Do not hesitate to each us via support for any questions.

Due to an incoming maintenance operation. We disabled the addon pulsar creation.

EDIT 20:57UTC - creation is enabled.

We identified issues on our metrics and accesslogs storage. We are working to fix the problem which is currently causing some difficulties on the query-side.

Users are experiencing HTTP errors on website heptapod.host.

** UPDATE ** 2022-03-24 15:40 UTC website does not have HTTP errors anymore

As announced, cellar-c1 has been definitively shutdown.

If you lack some files that were on it, please contact the support with all the informations: add-on ID, bucket name, etc.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 16/03/22 16:00 UTC for a 30 minutes window.

Our support team stays at your disposal for any questions.

We are currently having various networking issues (packet loss or slow response times) on our Paris infrastructure. We are investigating.

Some services are also impacted:

• Pulsar
• Metrics
• Access logs

EDIT 18:20 UTC: Our network provider is investigating the issue.

EDIT 18:28 UTC: The issue has been identified and has been escalated. Logs may also be impacted.

EDIT 18:44 UTC: The issue is still being worked out but Pulsar and Logs are now working fine again.

EDIT 19:26 UTC: The issue has been fixed by the network provider at 18:54 UTC. All components are now working fine again. Access logs are being ingested and may have some lag for a few hours. Sorry for the inconvenience.

We identified issues on our metrics/accesslogs storage. We are working to fix the problem which is currently causing some lags in the ingress data plane.

EDIT 12:04 UTC: The lag in the ingestion pipeline has been resolved.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 18/03/22 10:00 UTC for a 30 minutes window

Our support team stays at your disposal for any questions.

EDIT 11:00 UTC: The brownout has started and will last for 30 minutes.

EDIT 11:30 UTC: The brownout has ended. The service will be decommissioned next Monday.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 14/03/22 09:30 UTC for a 30 minutes window.

Our support team stays at your disposal for any questions.

EDIT 09:36 UTC: The brownout is starting. It will last for 30 minutes.

EDIT 10:07 UTC: The brownout has ended. Next one will happen on 16/03/22 16:00 UTC for a 30 minutes window.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 11/03/22 14:00 UTC for a 10 minutes window.

Our support team stays at your disposal for any questions.

EDIT 14:00 UTC: The brownout is starting and will last for 10 minutes.

EDIT 14:10 UTC: The brownout has ended. Next one will happen on 14/03/22 09:30 UTC for a 30 minutes window.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 09/03/22 10:00 UTC for a 10 minutes window.

Our support team stays at your disposal for any questions.

EDIT 10:00 UTC: The brownout has started.

EDIT 10:10 UTC: The brownout has ended. Next one will happen on 11/03/22 14:00 UTC

Our Cellar C1 cluster service has currently connectivity issues leading to failed requests. We are investigating with our network provider the reason of those issues.

Edit: Connectivity issues has been solved by our network provider. The service should run as expected

We identified issues on our metrics/accesslogs storage. We are working to fix the problem which is currently causing timeouts on queries.

EDIT 10:27 UTC: Queries have returned to normal, Metrics and Access logs should now be reachable. We are monitoring the queries.

EDIT 11:03 UTC: Queries have returned to normal, Metrics and Access logs should now be reachable.

An add-on reverse proxy started behaving erratically. This triggered timeouts and unreachability for some add-ons if the active connections were proxied through it. It has been restarted, which fixed the issue.

Sorry for the inconvenience.

Some of the databases hosted on that cluster were unreachable due to a node failure during a few hours. The problem has been fixed and the failure will be investigated further. Dedicated databases were not impacted.