Upcoming IP changes, and information about the recent attack

News concerning Priority Colo, this includes network status updates, upgrades to services, new services, and general stuff you might be interested in knowing.

Moderator: Admins

Post Reply
porcupine
Site Admin
Posts: 669
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Upcoming IP changes, and information about the recent attack

Post by porcupine » Thu Jul 10, 2003 2:53 am

Ok guys,

Well this should have been written earlier, but i've been swamped, and exhausted from the work.

As many had noticed, there were intermittent connectivity issues on 07/08/2003 in the early morning. This was the result of a large DDoS attack that had overwhelmed the upstreams routers. As a result, everytime the routers were brought back up, they overloaded and became unresponsive. Eventually the attacks were filtered, the ip's being assaulted were blacklisted, and the target was found.

Unfortunatly it seems the attackers were intent on shutting down demodemo.com (they sell flash tutorials for control panels for anyone who doesne't know them). The owner of DemoDemo.com has opted to have his sites moved to a server of his own on a private line, 100% independant of our connectivity. As a result, further attacks against Vito will be properly filtered outside our network, and things should return to normal basically.

In addition to this, while we submitted an ARIN request for our own IP zone thus we may stop outsourcing our bgp routing to someone larger, it was denied. In light of this we've arranged for another 100mbps circuit from Peer1 Networks, which will provide us with essentially a de-aggregated block of IP addresses, and will allow us to *completely* control our own routing on these blocks through our own bgp routing.

We will be providing each customer with IP addresses to renumber into. Basically the new IP's will have an added backup in comparison to the current IP addresses, and may also have slightly different routing configurations/peers. Both sets of IP addresses will remain 100% active while people are numbering over, customers will be provided several months to move their sites over to the new IP addresses.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com

porcupine
Site Admin
Posts: 669
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine » Thu Jul 10, 2003 10:00 am

In my haste, I missed a few things:

1. The main server IP address for www5 has been null routed (completely blocked) far upstream by Yipes!, Istop, etc., thus FTP on www5 will not work for a day or two, this is unavoidable, it will be unblocked once Vito (owner of DemoDemo.com) is completely moved to his new line.

2. A few of the lesser IP's on the www5 server (which only Vito was using) are also blocked across most/all links due to DDoS attempts.

3. In our haste to move any accounts that happened to have been created in error by us, or the customers on the main server ip on www5, we used the CPanel "move site ip address" feature, this did not properly redo the ftp.domain.com records based on how private shared IP addresses work (ftp.domain.com has to point to the main server ip address, the "change site ip address" tool points them to the ip you're changing them to), this will be corrected by hand (just takes a few as theres about 250 domains to do this to. Since the main server ip address is filtered anyhow, this makes little difference at this point.

4. Since the main server IP is filtered, you cannot view phpinfo. For CPanel/WHM logins, you can use www.yourdomain.com/cpanel/ and www.yourdomain.com/whm/ , customers are most definatly not locked out of their cpanels, only ftp.

And last, but most definatly not least, I'd like to extend my most sincere appologies to everyone. White Distributed Denial of Service attacks are like robbed, you can't exactly choose whether or not they happen to you, we do realise how much this has inconvinienced a lot of people. Personally speaking, the fact that these attacks are attempting to shutdown a customers legitimate business which is his bread and butter (puts food on his families table) makes it ten times worse. Anyone with any educated information of who is responsible for organizing the said attacks should most definatly come forward if they know anything.

Regards,
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com

porcupine
Site Admin
Posts: 669
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine » Sun Jul 13, 2003 4:19 pm

Just a update as many have asked.

The main server IP on www5 is still blocked.

Thus no FTP access still. I was hoping that Yipes! would have removed the null-route on the IP address, but they have not yet, so we must hope for monday (as the request was made late friday afternoon, probably when most of their techs were going home for the weekend.)

Should this not be resolved on their end by Monday, we'll have to find another way around this problem, eg. assigning a new main server ip address, etc.

On that note, some people may also find that email is bouncing when attemtping to send out. This unfortunatly is beyond our control, and is a result of remote mailservers doing some strange stuff (which i do not believe is RFC standard in any way shape or form). Basically mail goes out each resellers ip address, yet some servers are attempting to connect back to the main server ip (as it's listed as hostname in the email headers), when they can't, they reject the email. While this is 100% beyond our reasonable control, we hope to have the ip un-null-routed on the Yipes link before monday is over, eliminating this problem.

Thanks for your patience and understanding everyone :).

Regards,
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com

porcupine
Site Admin
Posts: 669
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine » Mon Jul 14, 2003 6:48 pm

Just got off the phone with Matthew from Istop/DCI.

He indicated that the nullroute/black-hole on the 66.11.162.63 IP address has been removed on the NAC link, and that Yipes! has responded, and it's in the "todo" queue. Apparently Yipes! has a policy of not doing any non-critical routing related tasks during the "business day", as a "just in case" (eg. so if someone botches it, it has less impact on their daytime customers, which likely is a good chunk of their revenue).

As a result, Yipes! should be removing the black-hole on that IP address at the end of their business day, (Which I believe is in the next 1.5 hours, not midnight, not sure what timezone they count it in).

After that, everything should return to 100% normal status! :D.
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com

porcupine
Site Admin
Posts: 669
Joined: Wed Jun 12, 2002 5:57 pm
Location: Toronto, Ontario
Contact:

Post by porcupine » Tue Jul 15, 2003 6:43 am

The main server ip (66.11.162.63) was unblocked awhile before midnight last night.

*ANYONE* still experiencing difficulties, errors, or problems in relation to this should email, icq, aim, helpdesk, etc. myself immediatly and we'll scan for any straggling problems :).
Myles Loosley-Millman
Priority Colo Inc.
myles@prioritycolo.com
http://www.prioritycolo.com

Post Reply