Sparked Host LLC - bak-par-01 Hardware failure – Incident details

Third Party: Cloudflare Network experiencing degraded performance

bak-par-01 Hardware failure

Monitoring
Degraded performance
Started 3 months ago

Affected

Locations

Degraded performance from 2:05 PM to 8:11 PM, Operational from 8:11 PM to 3:30 PM, Degraded performance from 3:30 PM to 12:00 AM

Paris

Degraded performance from 2:05 PM to 8:11 PM, Operational from 8:11 PM to 3:30 PM, Degraded performance from 3:30 PM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    A brand new chassis from a different vendor is scheduled to be installed today.

  • Update
    Update

    Update: The backup server in Paris had a full replacement done, and we're continuing to see I/O errors, indicating something lower-level, either in the hardware or file system, is causing writes and reads to the disks to fail.

    As a result, we're going to be taking the backup server offline for the moment until we can

    re-seat the disks and do additional hardware testing.

    Backups will be sent to another location until this can get sorted out.

  • Identified
    Identified

    We have identified a faulty component on a backup server in Paris, France that is affecting the following components:
    - Backup retrieval
    - Terminated service recovery operations

    A new chassis is en-route and we will swap disks into this new chassis when it arrives.