s26 Security Breach

Posts: 474 · Joined: 05 Feb 2004 Posts: 474

Approximately 12 hours ago we located a security breach on s26.sv.crucialx.net, upon investigation it was found that all files in format index* were replaced with a defaced webpage.

Upon further investigation we were able to find the cause of the breakin to the following 2.6 Linux Kernel Vulnerability: http://www.securityfocus.com/archive/1/440003/30/0/threaded

The exploit has now been removed, and system upgraded and patched to secure the vulnerability. We are still currently in the process of restoring the hacked files from backups.

Crucial Paradigm has very strict backup and security policies, this is the first time in 2 years that we have had an internal web hosting server hacked to gain root access. To ensure such an event does not happen again, we will be updating our upgrade policy on servers to do daily checks for vulnerable software, and update any vulnerable software each day.

All accounts should be restored within the next 24 hours, although most should be restored within the next few hours.

Please accept our sincere apologies for any inconvenience caused.

Kind Regards,
Aaron Weller
Crucial Paradigm

Posts: 474 · Joined: 05 Feb 2004 Posts: 474

We have had customers asking a wide range of questions, so I'm going to try and answer them all here.

Extent of the Security Breach:
The type of attack against our server is mass defacement, in this case the hacker replaced all index pages (home, index, admin, etc) with their own page. These pages included some of the cPanel and Fantastico system files, and is why when you login to cPanel some areas still show the hacked page. No databases, or other data should have been effected. Unfortunately due to our daily backups being stored on the same physical server as the sites are hosted on, our daily backups were corrupted as well. As a result we have been restoring hacked files from our weekly remote backups. This is a lenghtly process, and can take time - we can assure you that we are working on this 24/7 since it was detected.

How did they get in?
Due to a recent vulnerbility detected in the 2.6 Kernel, this allows a user to get full administrator (root) access to the server. Details can be found here: http://www.securityfocus.com/archive/1/440003/30/0/threaded
This was not a configuration issue where we left the server open to attack, but rather a vulnerability in the heart of the operating system. This vulnerability has since been patched, and the server is no longer vulnerable to attack.

Why is it taking so long?
The restoration process is quite lenghtly, the server has 100,000s of files on it which results in the process taking quite some time. Each file needs to be checked if it has been effected, then if it has the backup needs to be obtained from the backup server. If you require an urgent restoration of your account, please submit a ticket and we will attempt to do this for you, however due to the high number of tickets we are dealing with at this time there may be slight delay in completing this.

What is being done now?
Accounts are being restored, and scripts running on the server to find any infected files.

When will the restoration be complete?
Our intial estimations where a bit off, it will probably take another 24 hours to complete the restoration - during the first part of the restoration we were getting all index files from backup, and only found out recently that index files were not the only ones being effected - but also home, admin, among others. We have since written new restoration scripts which will take these into account, and we are working to restore all the damaged files.

What is being done to ensure this does not happen again?
We have made modifications to our server update policy, and we will now be doing daily updates of software on servers to ensure we have the latest software on all servers - it should be noted that we already had a system in place which we regularly updated software, which has worked very well for the past 2 years (with no servers hacked with root access during this time), however it is not as bulletproof as it could be.

Why haven't you replied to my ticket yet?
At this point we have 100s of tickets in our support queue, and we are working as quickly as possible to resolve all issues related with s26 - this can result in a delayed reply to your ticket.

If you wish to moved off s26 due to current and past problems, we can arrange this for you - please submit a ticket and ask to be transfered to a new server.

Please accept our apologies for the inconvenience caused during this outage, I'd like to extend my thanks to all customers who have been very patient, as well as tech support staff who have been working overtime to get this issue resolved.

Kind Regards,
Aaron Weller
Crucial Paradigm

Posts: 8 · Joined: 30 Jun 2005 Posts: 8 Location: Australia

The restoration of all hacked files will be completed in next few hours.
We apologies for the inconvenience caused.

Mahesh.

Posts: 8 · Joined: 30 Jun 2005 Posts: 8 Location: Australia

Hello,

This is to update that we have finished restoring all the hacked pages on the server.

Still if you are having issues with the hacked pages, then please open a new support ticket and we will take care of it at the earliest.