[AMS] Issues with EMK (Enterprise Managed Kubernetes)

Incident Report for Fuga Cloud

Postmortem

An unforseen issue followed after scheduled maintenance caused Kubernetes RBAC objects to not properly propogate throughout EMK clusters causing authorization errors with our cloud native networking infrastructure after midnight (CET timezone).

After forcing the propogation of these objects EMK clusters could recover. Workloads of EMK clusters continued to work unless altered after said time.

Posted Nov 20, 2023 - 13:02 CET

Resolved

This incident has been resolved.

Posted Nov 17, 2023 - 16:30 CET

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Nov 17, 2023 - 14:45 CET

Identified

The issue has been identified and a fix is being implemented.

Posted Nov 17, 2023 - 13:14 CET

Investigating

We are currently investigating this issue.

Posted Nov 17, 2023 - 10:30 CET

This incident affected: Cloud API Services AMS and Compute Services Availability Zones A-B-C AMS.