Sparked Host LLC - Multi POP Outage – Detalles del incidente

Sistemas funcionando con normalidad

Multi POP Outage

Resuelto
Rendimiento degradado
Iniciado el hace alrededor de 1 mesDuró 21 días

Afectado

Ubicaciones

En funcionamiento de 8:29 PM a 2:11 PM, Rendimiento degradado de 2:11 PM a 12:30 AM

Ciudad del lago salado

En funcionamiento de 8:29 PM a 2:11 PM, Rendimiento degradado de 2:11 PM a 12:30 AM

Actualizaciones
  • Resuelto
    Resuelto

    As of this time, all locations are stable and operating as expected. Our team will be deploying on-prem protection in Miami on a future date, which will be scheduled in advance.

  • Actualizar
    Actualizar

    We are currently experiencing issues affecting services routed through our Salt Lake City location. Our upstream provider utilizes mitigation capacity from Path Network. Path is presently encountering filtering and capacity limitations during volumetric events. This can cause packet loss, intermittent connection drops, increased latency, and short periods of instability for traffic passing through this region.

    These issues are upstream of our network. Our provider has already begun the process of replacing Path with a higher capacity mitigation solution. This replacement aims to restore full filtering capability, increase throughput headroom, and stabilize all attack handling in Salt Lake City.

    During this transition period, mitigation behavior may vary depending on the size and nature of incoming attacks. Our team is in active communication with the upstream provider to minimize impact while the new system is being integrated. As each stage of the upstream replacement is completed, customers should see noticeable improvements in stability and overall performance.

    We will continue to monitor the situation closely until the new mitigation platform is fully active. Further updates will be posted as progress continues.

  • Identificado
    Identificado

    We are adding Salt Lake City to the scope of this incident, as there have been intermittent periods of packet loss, due to large DDoS attacks resulting in degradation of network quality of our DDoS mitigiation upstream. We are continuing to work on solutions for this.

  • Supervisando
    Supervisando

    We have rolled back changes made to our network that were affecting authentication to Minecraft servers and other services.

  • Identificado
    Identificado

    Our team has identified instability in our network, specifically with connection failures to services like Mojang's Authentication servers, discord APIs, etc. Our upstream is in the process of working on what is causing this. We will provide an update soon.

  • Actualizar
    Actualizar

    Our team has onboarded a new protection provider in Miami and is monitoring the results. There may be brief connectivity blips while the process occurs.

  • Supervisando
    Supervisando

    Services in Miami and Paris are currently online and operating normally. Earlier, we experienced a sudden connectivity loss across both POPs. Our team promptly switched to our backup transit provider, restoring service.

    Additionally, our DDoS mitigation provider in Ashburn reported a brief period of elevated latency and packet loss, which has since stabilized.

    We’re continuing to investigate the root cause and are in contact with our upstream NOCs for further information. We’ll provide updates as they become available.

  • Investigando
    Investigando

    Our monitoring observed and alerted on packet loss and large traffic drops in multiple locations.