VPSRENTO
AMS1 core switch failure — degraded networking, 4h12m
Summary. At 03:17 UTC, the primary top-of-rack switch in our AMS1 rack stopped forwarding traffic on 22 of 48 ports. Roughly 40% of Amsterdam VPS instances lost external connectivity; the remainder saw elevated latency and packet loss on the redundant path. All services were fully restored by 07:29 UTC. No data was lost and no storage layer was affected.
Root cause. A failing ASIC on the primary switch's line card, triggered by a firmware bug in the buffer-management unit (vendor advisory PSIRT-2026-041, published after our incident). Automatic failover to the secondary switch engaged, but a misconfigured LACP hash on one aggregation group left 22 ports pinned to the dead unit.
Remediation. The faulty line card was replaced from on-site spares at 07:12 UTC and the LACP groups rebalanced. We have since applied the vendor firmware patch on all AMS1 and ZRH1 switches, added a quarterly automated failover drill that verifies every port group rather than just the master link, and deployed per-port forwarding telemetry so a partially dead switch pages us within 60 seconds instead of relying on aggregate link state.
Credits. The event breached our 99.9% monthly SLA for affected Amsterdam customers. SLA credits were applied automatically to all AMS1 services active during the window — no ticket required, per our SLA.
- 03:17 UTC — Monitoring flags packet loss across 3 AMS1 hypervisors. On-call paged.
- 03:24 UTC — Confirmed: primary ToR switch not forwarding on 22/48 ports. Failover partially engaged.
- 03:41 UTC — Status page updated to "degraded performance" for Netherlands. Traffic shifted to secondary path.
- 05:02 UTC — Datacenter remote hands dispatched to rack for line-card replacement.
- 07:12 UTC — Line card replaced, LACP groups rebalanced, full forwarding restored.
- 07:29 UTC — All checks green. Incident marked resolved. Total impact: 4h12m.
- 2026-06-16 — Firmware patch validated in staging, rolled out to AMS1 and ZRH1.
- 2026-06-21 — This retrospective published. SLA credits applied.