07-29-2003 Continuing Network Problems

Post by **porcupine** » Wed Jul 30, 2003 12:22 am

Hi guys,

If you're reading this, first off, thanks for your patience, we really appreciate it. Some of you are just new with us, some of you have been with us for 1+ years now, and know that prior to this month, all was smooth sailing.

Currently our upstream (Istop) is having problems with their bgp sessions dropping between a set of routers. They've been working on it for some time, and basically are having major problems with this.

On the news of the additional feed, as (previously written in nearby threads), Istop's stability recently has been questionable... For a status update, we've got the independant feed to Peer1 active, we have the IP ranges, and we have the additional router for the task. Basically the problem being, that the old IP addresses will not be routed through it, thus only after IP's are moved will there be complete stability. Currently we're working with Istop, coming to an agreement to allow our current IP's to be announced over both links, which will result in an immediate improovement once this is done.

Moving our sessions from Istop's direct route to a BGP configuration will require reconfiguring our current main router (R1), and will result in approximately 15 minutes of downtime, which we will note ahead of time in this (or another appropriately titled) thread. Once this has been accomplished, these issues should essentially vanish.

Our goal is to have the current IP's announced over both links by Monday August 4th, and have the new IP addresses up, active, and ready for migration by Thursday August 7th.

During this time, both sets of IP addresses will be active, and routed to your servers. Any Hassles of DNS propogation will not be present during this time, as the content will not be moving, the servers are not moving, the IP's are changing, but both the new, and old IP addresses will connect to relavent content, making a smooth, and easy going transition for all clients. Ideally we'd like to see the vast majority of clients moved to the new IP addresses within 1-2 months, but will keep both routed properly for as long as it takes to get the job done conviniently, comfortably, and properly for ourselves, and our clients.

Once again, thank you everyone for your patience, you have my word that we're doing everything within our reasonable power to get this rolling, and ironed out as smoothly as possible.

Regards,

Post by **porcupine** » Wed Jul 30, 2003 1:07 am

Incidentally, this forum is not locked, nor this thread, if you have any comments, questions, suggestions, etc. etc. please let them be heard

.

Alexander · Post by **Alexander** » Wed Jul 30, 2003 6:01 am

Would this mean all reseller ip's change aswell?

Post by **porcupine** » Wed Jul 30, 2003 8:42 am

Alexander wrote:Would this mean all reseller ip's change aswell?

Over time, Yes, this would mean the resellers IP addresses would change as well. The resellers would be in the same situation where they would not have to play any of the propogation games while changing ip's, no downtime, or stuff pointing to the wrong locations as both new and old IP addresses would be actively routing to the reseller servers.

Alexander · Post by **Alexander** » Wed Jul 30, 2003 9:02 am

If you eventually could post a "old ip -> new ip" list that'd be great. Glad to hear the service won't be interupted at all

Post by **porcupine** » Wed Jul 30, 2003 9:21 am

Alexander wrote:If you eventually could post a "old ip -> new ip" list that'd be great. Glad to hear the service won't be interupted at all

Wont be necessary. Each customer will be individually contacted with their new ip's. That and we don't want anyone mass harvesting the IP information, you never know what might come of something like that (bad idea security wise to post a massive list).

Alexander · Post by **Alexander** » Wed Jul 30, 2003 9:46 am

OK, only need my own info anyways

Great work miles!

Post by **porcupine** » Fri Aug 01, 2003 9:49 pm

Just an update for everyone waiting, and reading

.

IStop will permit us to BGP advertise the current IP blocks over both links (as they'd been pushing for us to use BGP for some time, as several of the gameservers on our network transmit a high amount of packets per second, creating a relatively large load on their routers).

This means that the current IP addresses will become more redundant the second we get both links BGP'd on our end.

ARIN has received our paperwork (the request for the ASN), and has approved it, as well as received, and acknowledged payment. We've faxed in the agreement papers, and are simply waiting for the ASN. Unfortunatly without the ASN, we cannot bring up the BGP sessions.

We've got R2 online, and setup a test BGP session with 2 other peers using a private test ASN, everything on that scale is working properly.

Because we still dont have the ASN from ARIN (and they dont work weekends), i'm uncertain if this compromises our monday estimate, as we don't know what time monday we'll get it, nor do we know if the other providers involved (Istop/Peer1) will be able to get BGP up on their ends, and configured in such short notice.

Fortunatly since the new IP's, and the old IP's will be broadcasted using the BGP4 protocol, this makes the move less rushed for most. Naturally we will still be migrating to the new IP addresses, but this should take the edge off the stress, for most, and elongate the migration period.

With the addition of the ASN, we will be making a few modifications in relation to TorIX, basically directly peering with several providers, opposed to using the Istop peering arrangements.

Post by **porcupine** » Mon Aug 04, 2003 12:56 pm

Ok,

Well it's 1:00pm on Monday. I'm uncertain whether ARIN will have the ASN to us today at all frankly (they said they did, but I forgot today is a civic holiday in Canada).

Since it seems that Peer1, and Istop, neither of which have their general offices open today, the BGP session definatly will not be going up that fast unfortunatly. We'll get back on track tomorrow hopefully, and have the ASN before mid-day.

Some people may have experienced hiccups in the network within the past 24-48 hours. This was due to the Yipes POP down in Manhattan, they reported having routing issues. The hiccups were the result of transit routing through Istop's alternate providers.

As some people have asked, the default timeout on BGP sessions is 180 seconds. As such, when a session fails, unlike the common myth, traffic is not instantly re-routed, it may take as long as 3 minutes to properly establish a new path. This is simply the way that BGP works, while the timeout can be reduced, when a link goes down, traffic needs to find a new path, hope that clears it up for anyone who wondered why there were brief hiccups

.

Post by **porcupine** » Wed Aug 06, 2003 2:15 am

Ok,

Well todays update is as follows:

ARIN Has allocated the ASN, no problems there.

Router2 is online, and configured. We're simply waiting for Peer1 to turn up their BGP session to start the testing directly with Peer1.

Istop is insisting that if we're to do BGP over this link, it should be to their new rack/switch in our suite (instead of the old one which they intend to migrate all customers off of), we've put in a work order to get the additional cross connect run to this, and it should be done soon, which will kill two birds with one stone.

Initial BGP testing should go up tomorrow on the still inactive link thus we may prepare, test, and perfect the still idle link before touching the production level one.

Post by **porcupine** » Wed Aug 06, 2003 8:06 pm

Ok guys,

First off, I'm going to split this topic now, as it's kinda outgrown it's original intent, and we want people reading on the right topic.

The peer1 link is up and active, BGP is running, properly, without issue over the peer1 link and advertising the peer1 ip's to the internet. Connectivity has been tested, and everything appears to be in order thus far.

Next comes adding the current IP's from Istop to this route announcement, and failing over the Istop IP's to the peer1 link (R2) while we reconfigure R1 to do BGP with Istop.

We will post a schedule when we have it down in concrete when we're going to bring this session online, should be soon though at this pace.

Post by **porcupine** » Fri Aug 08, 2003 9:23 am

The BGP upgrade has been scheduled.

It will be done on Saturday August 9th, the maintenance window is between 12:30am and 4:30am.

We expect several small "hiccups" as routes are reconfigured, disabled, reconverge, and brought back up. This will be one (if not) the final configuration change involving any downtime. Downtime during this time is expected not to exceed 30 minutes of cummulative/combined downtime for the entire maintenance window, but as noted, will involve several short 'hiccups'.

I will be posting another thread to grab peoples attention on this momentarily, so that everyone gets some notice of the maintenance.

Post by **porcupine** » Fri Aug 22, 2003 5:01 pm

Ok,

Just in answer to anyone whose not asked, but been wondering (or people who've asked, but not quite so recently), I've had a few people say "where are the new IP addresses?". Well the IP's are here, everything in the shared IP's has been provisioned out, new accounts are being setup on them somewhat, and the resellers are being setup on them.

Why haven't you gotten them yet? Well, as i'm sure you can imagine, we've been extrodinarily busy with all the issues the power failures have brought up, but thats not the main reason. Peer1 gave us the new blocks of IP addresses, and while only one IP had been abused, the entire block had been listed in spews. While it's usually a total pain to get IP's removed from spews.org and the other anti-spam listings, there was only one listed ip address, and we don't have much of a choice. Problem being right now, it looks like the spammers have been DDoS'ing many of the major anti-spam websites, anyone whose attempt to visit www.spews.org in the past few days will have noticed it's down.

Until spews is up and kicking again, chances of getting de-listed are next to null, thus the new IP's will not be dolled out until then (except by specific request). The current IP's as previously mentioned are fully advertised through our BGP session, thus they are currently just as redundant as the new ip blocks, simply not spews listed.