www5 Server (Unscheduled Downtime)

Post by **porcupine** » Sun May 11, 2003 6:07 am

It would seem that www5 has been compromised. We have shut the server down and are reinstalling the OS and the user data from backups. Mysql data, and more recent user data will be pulled from the working directories as little harm can come from that. We have shut the server down until it's re-prepared to prevent the compromised system from giving up any amount of user data in its current state.

We will post updates as they come along.

Post by **porcupine** » Sun May 11, 2003 11:17 am

ok people,

Well it's taken what seems an eternity, but here's the scoop:

www5 is fully reloaded on a new hard drive (faster too i might add, 8mb of cache), and loaded from scratch. *MOST* of the content was recovered straight from the old drive (eg. fresh data), some had to be recovered from the backups. Any accounts that were activated within the past 5 days most likely do not exist on the new system, because we lost our /etc directory on the old drive (possibly deleted, possibly lost due to filesystem errors of our improper shutdown, unknown), but as a result, any content regularly stored in /etc (account usernames/passwords/quotas, etc.) were recovered from a backup done 5 days ago, thus any new accounts added since, likely have their data intact, but no account!

If you've got end users in this situation give me a yelp, and i'll look into them and fish out their data

.

Anyhow the server is doing what should be it's final boot right now, and should be back up with 95-98% of the sites functional and back online with little/no data loss as a result of this malicious attack.

Post by **porcupine** » Sun May 11, 2003 11:41 am

Just an update, FSCK (File System Checker) is running across the drives, and the old drive has forced a check (which unfortunatly takes awhile). in the interest of safety and preparedness, we're keeping the second drive onboard, just in case some data didn't move cleanly, but as a result, we have to wait for this scanning sequence to finish before the server will fully boot.

vito · Post by **vito** » Sun May 11, 2003 12:03 pm

Any projected timeline?

Vito

Post by **porcupine** » Sun May 11, 2003 1:23 pm

Sorry didn't get back to update this earlier, yours was one of the last sites to come back up, should be back in it's entirety now Vito, as of ~ 1.5 hours ago.

jk1 · Post by **jk1** » Sun May 11, 2003 2:28 pm

Thanks for keeping us informed, Myles.

Waiting for cpanel & email to be restored and then I should be set!

Post by **porcupine** » Sun May 11, 2003 8:56 pm

Ok well the last of the backups have completed, and the "fix" type of scripts are running to fix any quota, email, permission, etc. problems.

The following accounts couldn't be restored normally, and will be restored (by us, unless otherwise notified) by hand:

Security violation.... (homedir [/home/upperlof] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/titanhyp] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/snowblin] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/legoau] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/insbird] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/engraved] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/dzyntech] does not exist)Account Restore Failed...

Post by **porcupine** » Sun May 11, 2003 9:47 pm

The afforementioned accounts have all been added.

Maintenance should now be complete, and the server back at 100%. If anyone has *ANY* problems which didn't exist before (big, or small), please make sure to let us know. If anyones site is having critical problems, please keep in mind our emergency-pager if you classify it reasonably as an emergency.

Misha · Post by **Misha** » Mon May 12, 2003 2:05 pm

Thanks Myles. Great job