| Author |
Message |
| < Public Announcements ~ s26 Hardware Failure |
|
 Posted:
Mon Apr 03, 2006 11:58 am
|
|
|
|
|
At approximately 15:45 CST on Sunday 2nd of April s26.sv.crucialx.net experienced a hard drive failure in the primary RAID array. As soon as the drive failure was detected we started work on attempting to restore the array - unfortunately this was unsuccessful and we are now replacing the failed drive and starting to restore accounts from our daily backups.
Due to the number of accounts which need to be restored (over 600) this is lengthy process, and could take up to 24-48 hours to complete restoration of all accounts.
Please note we DO have standby servers readily available within minutes, but due the type of failure (hard drive) it is not possible to restore the accounts instantly on a standby server, and requires each account to be individually restored from backups - which can take some time.
If you wish to receive an SLA refund due to the downtime caused - please do so AFTER your account has been restored. Be sure to include your domain name, and time period your site was down. Note: Our SLA currently offers 99.9% NETWORK uptime guarantee, and this specific case is hardware failure which is not directly covered by our SLA - however all SLA refund requests will be fairly reviewed.
We sincerely apologise for any inconvenience caused, and you can be assured that we are working on 24/7 until it has been resolved. Please feel free to contact us if you have any questions or concerns.
Kind Regards,
Aaron Weller
Crucial Paradigm |
|
|
|
|
|
 |
|
| Posted:
Tue Apr 04, 2006 6:31 pm
|
|
|
|
|
UPDATE:
We have restored approximately 390 accounts at this stage and working as fast as possible to continue restoring the remainder (210) of the accounts.
A few things to keep in mind:
- We working 24/7 on this issue, and have a number of staff members working to get all accounts restored as quickly as possible. Due to the process of restoring accounts quicker, we could not complete it faster if we had 5 staff members working on it or 100 staff members working on it.
- We DO use quality hardware, including quality hard drives, and hardware based RAID. However, hard drive crashes, and data corruption are inevitable at times, and this is why we keep 3 sets of backups, one of which are stored offsite. We also use quality SCSI server grade hard drives, which come with a 5 year warrenty, however they are still prone to failure.
- We do have multiple standby servers available, but restoring accounts to standby servers would take LONGER than if we restore them on original servers, and would require name servers changes. The standby servers are there in case of failure of any part of the except for the RAID Array/Hard Drives.
If your account has been restored, but is not working correctly, please submit a support ticket and will work to fix any problems you may have ASAP.
Please feel free to contact us if you have any questions or concerns.
Kind Regards,
Aaron Weller
Crucial Paradigm |
|
|
|
|
|
|
|
| Posted:
Tue Apr 04, 2006 11:18 pm
|
|
|
|
|
UPDATE:
Approximately 560 accounts have now been restored. |
|
|
|
|
|
|
|
| Posted:
Thu Apr 06, 2006 6:19 pm
|
|
|
|
|
The restoration was completed some time ago, if you have any problems with your account please submit a support ticket, and we will look into it ASAP.
I'd like to thank all our customers for their patience during the restoration process as well as our staff members who have worked overtime to make sure that s26 was restored as quickly as possible.
Please feel free to contact us if you have any questions or concerns.
Kind Regards,
Aaron Weller
Crucial Paradigm |
|
|
|
|
|
|
|
