Incident Report: AU 17/Sep/2010 5PM AEST (+10GMT) - s392.au.crucialx.net

[Aaron]

You are being contacted as you have a reseller account on the server s392.au.crucialx.net.

EVENT START DATE: 17/Sep/2010
EVENT START TIME: 5PM AEST
EVENT END DATE: n/a
EVENT END TIME: n/a

EVENT IMPACT: Customers with reseller accounts on s392.au.crucialx.net.

EVENT DETAILS: A security incident has been detected on this server, and some customers sites have been effected. As a result we will be restoring all accounts from our daily backups.

While a forensic investigation is underway, our highest priority is restore all customer accounts from backups. A backup will first be taken of all sites in their current state before we start the restoration. During the restoration your site will be offline for a period of 2-8 hours. During this time we ask you to refrain from submitting tickets until you receive notification that all accounts have been restored.

You will receive a notification just before we start the restoration which we expect to take place in 2-4 hours from now.

NOTE: Crucial Paradigm takes security very seriously and has a large number of measures we use to ensure our customers data stays secure. We will be completing a thorough investigation into why this security incident occurred, and to help prevent such an issue from occurring in future.

UPDATE (18/Sep/2010 1:07PM):
The backup of s392 has been completed, and we will be taking it offline shortly to do an OS re-install, and restore accounts from our backups.

This process will result in your site being offline for between 3-9 hours while we complete the re-installation of the server, and then restoration of all the accounts from backups.

[Aaron]

The backup is taking a little longer than expected, we will complete the restoration from backups once the backups are complete. We expect another few hours before this occurs.

[sanooj.m]

The backup is still running. The backups is being generated in the alphbetical order of usernames. It has now been completed till "g". We will keep you updated here.

[Gopeekrishnan]

We have reached accounts starting with 'U', expecting this to be finished in few hours.

Gopeekrishnan
Crucial Paradigm

[cybercomp]

any updates on this yet?

[cybercomp]

my account starts with "c" and it is still showing the muslim hacked stuff....

[cybercomp]

if anyone is interested, it seems to be CMS/DB type sites that have been affected on my account. appears to be an index.html file that has been injected, once I delete/rename that file the index.php is active again and all good (as far as I can tell)

edit - has also been sites with index.html files. just restored from my own backup or replaced with a maintenance message. at least there is no sign of being hacked now.

[Gopeekrishnan]

We are taking the backup of accounts starting by 'V'
Mainly the index files of accounts are the affected ones. Any how as a matter of security we will be doing an OS reload after wiping the whole content.

Regards
Gopeekrishnan
Crucial Paradigm

[cybercomp]

and to top is off the USA server account I use s369.c4.crucialx.net is down.. what a day!

[Aaron]

Originally Posted by cybercomp

and to top is off the USA server account I use s369.c4.crucialx.net is down.. what a day!

Yes, unfortunately it is. The server went down after we applied a temporary fix to ensure the same thing that happened to s392 does not happen to s369. This worked without any issues on all our servers except for s369. s369 crashed and instead of rebooting and bringing the server online we have had to go through several fscks (unrelated to the original incident).

The specific security issue we experienced was due to a 0day exploit (meaning no fix had been released yet). We managed to find a temporary fix to protect our servers until an official kernel update has been released. We will be providing a full report once the issue has been resolved.