Stoat Status Machine

Our domain has been successfully removed from the Spamhaus DBL blocklist. Email deliverability has been restored, and messages should no longer be rejected.

However, due to a backlog of queued messages, our email service provider is still processing a higher-than-normal volume of mail. As a result, some emails may experience delays of approximately 20 minutes.

We will continue to monitor the situation and provide further updates as needed.

We are aware of mail deliverability issues, we are currently listed on Spamhaus DBL and are trying to actively resolve the issue.

Some emails may also be delayed, this is likely due to the sheer volume of requests going to our service provider. We have mitigated this by increasing verification email expiry.

Just saw a burst of users coming online, attempting to re-scale.
Update: Scaled, monitoring performance, we may have to go
Update: Trying to push throughput even further
Update: We're at an architectural limit, going to temporarily disable some events (incl. typing indicators and user updates) to help ease congestion while an actual fix is being put together

I suspect we are hitting limits with our message pubsub, a solution is being put together.
Update: Scaled vertically for now.

Production services are now scaled up.
There is a possibility of hitting further bottlenecks but we should be okay for a moment.
Will continue to monitor and improve the deployment pattern.

Ordered more servers, waiting for fulfillment. Service is generally stable right now, but more load is expected either today or tomorrow peak hours.

Single-node cluster deployment was successful, now scaling it up.

Deployment is taking a little longer than expected, but I would expect this to take less than an hour to resolve.

We are currently working on scaling up our services.

Issue appears to be resolved. We are continuing to monitor for further issues.

We are monitoring the issue.

Track external service provider incident here: https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

The issue appears to be caused by an external networking services provider.

We are aware of occasional connectivity issues and are pinpointing the cause.

Maintenance is complete, all services are back online.

Services will be temporarily unavailable during this maintenance period.
Expect resolution within the hour.

The external provider has fixed the problem. We will continue to monitor the situation, however emails should now be sent in a timely manner.

The issue appears to stem from an external service provider. We are continuously monitoring the situation.

Some users are experiencing delays in receiving verification emails, for example when creating an account.