02-Sep-2004 - WWW3 Apache Problems - 7:30pm - 9:30pm

Announcements concerning Networking & Related News, Planned Outages, Anything which may affect your services.

Moderator: Admins

Post Reply
porcupine
Site Admin
Posts: 703
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

02-Sep-2004 - WWW3 Apache Problems - 7:30pm - 9:30pm

Post by porcupine »

Hi guys,

Customers, and users of the www3 server may have noticed some connectivity issues, specifically to httpd within the past two hours.

HTTPD has been up and down, due to a user script overloading the server. We were not able to isolate this script, as it was using up to 200mb/sec of memory at random on randon apache processes. By the time we ran the "apache-server-status" to check which files/pages were currently being accessed, apache was already dead in most cases.

Unfortunatly these types of problems are unavoiable when you have an environment such as the one which we proudly support, where customers are permitted to run the vast majority of scripts that they may code themselves, pay for, or download free, at their own will without severe restrictions. As such, every once in a while, a user will mistakenly (or maliciously) create a script that will inadversely disrupt server operations. While it is fairly rare for this to happen, as there are safeguards to "kill" off scripts that operate like this, inevitably some do eventually get through, this being one of them.

Due to the circumstance, and the speed at which the given script was utilizing available memory, and swap space, we were unable to isolate it. Instead we suspended accounts which were being accessed at the same time as the memory usage was rapidly climbing (as fast as 200mbytes/second). With some simple trial and error, we have isolated the user whose account was hosting the said script, and everything has returned to normal.

You may now return to your regular enjoyment of your browsing experience. We appologise for any inconviniences this may have caused.

Regards,
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
Post Reply