[AMS] Network incident

Incident Report for Cyso Cloud

Postmortem

A sudden accumulation of problems in the handling of QoS rules on virtual networking resulted in a disruption of public traffic in region AMS2. Both incoming and outgoing traffic on a virtual server, virtual router or load-balancer was not possible. This also impacted our internal and external services such as the EMK API and the customer portal. All services recovered after neutralizing the QoS rules and redeployment of network agent across the region.

Posted Oct 30, 2025 - 11:57 CET

Resolved

This incident has been resolved.

Posted Oct 29, 2025 - 09:02 CET

Monitoring

A fix has been implemented, and the last systems are recovering. We are monitoring the results.

Posted Oct 28, 2025 - 23:44 CET

Update

We have implemented an additional fix, and seeing further recovery in our virtual networking and EMK services. We are continuing to work on making all our services fully healthy again.

Posted Oct 28, 2025 - 23:24 CET

Update

We are continuing to work on a fix for this issue.

Posted Oct 28, 2025 - 22:57 CET

Update

We are continuing to work on implementing a fix for the diagnosed issue in our virtual networking services.

Posted Oct 28, 2025 - 22:12 CET

Update

Most services have been restored and are operating normally. We're actively working to resolve remaining issues affecting primarily Managed Kubernetes services. Our team is working hard on resolving these issues.

Thank you for your patience.

Posted Oct 28, 2025 - 21:05 CET

Update

We are working hard on implementing a fix, we are seeing some initial improvement in service availability.

Posted Oct 28, 2025 - 19:23 CET

Update

We are continuing to work on a fix for this issue.

Posted Oct 28, 2025 - 18:37 CET

Identified

The issue has been identified and a fix is being implemented.

Posted Oct 28, 2025 - 17:36 CET

Investigating

Some EMK instances suffer network connectivity issues. We are investigating the issue.

Posted Oct 28, 2025 - 17:08 CET

This incident affected: Portal and Dashboard Services, Website - Support Services, Backend API Services, Loadbalancer Services, Cloud API Services, Compute Services Availability Zones A/B/C, Virtual Networking Services, and Enterprise Managed Kubernetes Services.