Good news everybody!

First though, some bad news. It was bad news that it took us so long (possibly as long as 3 weeks) to really recognize this was a system-wide problem and to begin to narrow down the cause.

The good news is, we believe the slowness affecting some servers has been fixed for good as of right now!

It turned out that the ether channel between our core1 and core2 routers was maxing out at 1Gbps. This was weird to us because it should have been able to hit 3GBps, but for some reason wasn't.. and it wasn't dropping packets either like you'd normally see when a port becomes saturated.

It turned out the router card itself could only handle 1Gbps, even though it had 3 gige links going out of it! And, it was just queueing up packets rather than dropping them.. resulting in slowness overall for people on servers using core2 as their primary router.

We now switched to using our fiber channel cards which DO support over 1Gbps and things look okay. Right now it's doing about 1.6Gbps.

We're really sorry about this ongoing issue, and we're really happy it should be fixed now! If anybody could verify things seem better now from their side it'd make us feel more secure about our fix too!

Thanks!

josh!