www2 Maintenance - 01-Apr-2004 - 6:00pm - 2:00am

Announcements concerning Networking & Related News, Planned Outages, Anything which may affect your services.

Moderator: Admins

Post Reply
porcupine
Site Admin
Posts: 704
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

www2 Maintenance - 01-Apr-2004 - 6:00pm - 2:00am

Post by porcupine »

Hey guys,

Our www2 reseller server will be brought down for maintenace on the 1st of April, 2004 (no, not an April fools joke) for routine maintenance. Downtime is expected to be 5-10 minutes, during this 8 hour window. We will try to push this back as far as possible, we will be adding another 1/2 GB of RAM to help alleviate recent speed issues in relation to the many, many MySQL processes the server handles.

Let me know if you have any questions, as always :).

Regards,
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
porcupine
Site Admin
Posts: 704
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine »

Ok, well that upgrade certainly went anything but smoothly.

Heres the summary of what happened:

- www2 was gracefully shutdown, 512MB of DDR ECC Registered RAM was added
- www2 came back up, ran for about 15 minutes, then pagefaulted (crashed)
- RAM was moved to another slot, www2 came back up, ran for about 30 minutes. It was noticed that apache/php was broken from the first attempt (still uncertain as to how), specifically one of the php libraries, php was disabled as apache was recompiled
- Recompile finished, no impact, the server then crashed
- RAM was decided to be bad after 2 failures in a row, removed, and labeled as DOA. Server was rebooted without it only to find both apache and php non-functional
- Last few hours have been spent debugging this, currently apache is up, php is not. Basic apache compilations even failed, resulting in continual segfaults (aka no webserver).

This issue *will* be resolved before the sun is up, hopefully before midnight.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
porcupine
Site Admin
Posts: 704
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine »

As promised, php has been repaired.

It would appear that the error, though it was writing nothing to the /usr/local/apache/logs/error_log was being caused by php, more specifically, the Zend Optimizer portion, or its libraries.

After installing a fresh copy of Zend Optimizer based on a hunch, apache has restarted flawlessly with php support.

For future notice (and anyone searching the forum if they get stuck down the road), the following error *was* fixed by reinstalling zend optimizer, not repairing apache/php/etc:

[Thu Apr 1 23:28:31 2004] [warn] NameVirtualHost 66.199.181.199:80 has no VirtualHosts
/usr/local/apache/bin/apachectl: line 192: 7210 Segmentation fault (core dumped) $HTTPD -DSSL
/usr/local/apache/bin/apachectl startssl: httpd could not be started

If anyone has any comments, questions, or concerns, please do *not* hesitate to email me immediatly regarding this matter, or any related matter. Anyone whose account has not been kept up to our 99.5% uptime guarantee as a result of this should immediatly contact me regarding this also. Everyone affected by this has my appologies, quick hardware swap gone bad would be a fitting title to this post.

Sincerely,
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com
Post Reply