www5 Server (Unscheduled Downtime)

Announcements concerning Networking & Related News, Planned Outages, Anything which may affect your services.

Moderator: Admins

Post Reply
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

www5 Server (Unscheduled Downtime)

Post by porcupine »

It would seem that www5 has been compromised. We have shut the server down and are reinstalling the OS and the user data from backups. Mysql data, and more recent user data will be pulled from the working directories as little harm can come from that. We have shut the server down until it's re-prepared to prevent the compromised system from giving up any amount of user data in its current state.

We will post updates as they come along.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine »

ok people,

Well it's taken what seems an eternity, but here's the scoop:

www5 is fully reloaded on a new hard drive (faster too i might add, 8mb of cache), and loaded from scratch. *MOST* of the content was recovered straight from the old drive (eg. fresh data), some had to be recovered from the backups. Any accounts that were activated within the past 5 days most likely do not exist on the new system, because we lost our /etc directory on the old drive (possibly deleted, possibly lost due to filesystem errors of our improper shutdown, unknown), but as a result, any content regularly stored in /etc (account usernames/passwords/quotas, etc.) were recovered from a backup done 5 days ago, thus any new accounts added since, likely have their data intact, but no account!

If you've got end users in this situation give me a yelp, and i'll look into them and fish out their data :).

Anyhow the server is doing what should be it's final boot right now, and should be back up with 95-98% of the sites functional and back online with little/no data loss as a result of this malicious attack.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine »

Just an update, FSCK (File System Checker) is running across the drives, and the old drive has forced a check (which unfortunatly takes awhile). in the interest of safety and preparedness, we're keeping the second drive onboard, just in case some data didn't move cleanly, but as a result, we have to wait for this scanning sequence to finish before the server will fully boot.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
vito
newbie
Posts: 14
Joined: Mon Aug 05, 2002 9:30 pm
Location: Toronto, Canada
Contact:

Post by vito »

Any projected timeline?

Vito
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine »

Sorry didn't get back to update this earlier, yours was one of the last sites to come back up, should be back in it's entirety now Vito, as of ~ 1.5 hours ago.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
jk1
newbie
Posts: 3
Joined: Thu Oct 31, 2002 1:12 pm

Post by jk1 »

Thanks for keeping us informed, Myles.

Waiting for cpanel & email to be restored and then I should be set! :)
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine »

Ok well the last of the backups have completed, and the "fix" type of scripts are running to fix any quota, email, permission, etc. problems.

The following accounts couldn't be restored normally, and will be restored (by us, unless otherwise notified) by hand:

Security violation.... (homedir [/home/upperlof] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/titanhyp] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/snowblin] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/legoau] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/insbird] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/engraved] does not exist)Account Restore Failed...
Security violation.... (homedir [/home/dzyntech] does not exist)Account Restore Failed...
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine »

The afforementioned accounts have all been added.

Maintenance should now be complete, and the server back at 100%. If anyone has *ANY* problems which didn't exist before (big, or small), please make sure to let us know. If anyones site is having critical problems, please keep in mind our emergency-pager if you classify it reasonably as an emergency.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
Misha
newbie
Posts: 15
Joined: Sat Apr 26, 2003 10:09 pm

Post by Misha »

Thanks Myles. Great job
Post Reply