We're getting about 500k packets per second, 500mbps to 1.5gbps peak, it's a synflood from a botnet. Typically we can IP hop and null-route the old IP's. That usually buys us about a day until the botnet phones home to get the new IP's, at which point we just hop again. Since our DNS TTL is only 5 minutes at most we are down 5 minutes.
TODAY, the attackers hopped IP's to our new IP immediately. So they appear to be learning. But then again, so are we.
Gigenet's anti-DDoS service has helped us a ton here and is now serving as our front door IP to block the synflood. They've been really responsive.
Well, not following the RFCs is bad, but I can see why they do it.
Most people using low TTLs probably don't know what they're doing, and if you're a big ISP having to constantly make recursive queries hurts page load times for your customers, who'll blame you.
I really wish setting DNS TTL's to 5 minutes meant that browsers and everything else (OS caching) would respect that but sadly in my experience it doesn't.
This is hot off the heels of an email sent out last night:
As you’re no doubt aware, Posterous has had a rocky six days.
On Wednesday and Friday, our servers were hit by massive Denial of Service (DoS) attacks. We responded quickly and got back online within an hour, but it didn’t matter; the site went down and our users couldn’t post.
On Friday night, our team worked around the clock to move to new data centers, better capable of handling the onslaught. It wasn’t easy. Throughout the weekend we were fixing issues, optimizing the site, some things going smoothly, others less so.
Just at the moments we thought the worst was behind us, we’d run up against another challenge. It tested not only our technical abilities, but our stamina, patience, and we lost more than a few hairs in the process.
I’m happy to report Posterous is at 100% and better than ever. Switching to a new data center will help us avoid the type of attacks we saw last week, and the new, bigger, beefier servers will speed up the site and increase capacity. We were hit pretty hard, but we’ve come out stronger in the end.
While we were certainly frustrated, we know that no one was more frustrated than you. Your website was down, and I humbly apologize for that. Know that throughout these six days, restoring your site and your trust has been our number one priority.
At the same time, I have never been prouder of the Posterous team. Garry led the team in the server migration, Vince worked like a mad man until he passed out on his desk, and Jackson helped the community understand the impacts of some pretty technical issues. This week has been all hands on deck, and we had some pretty great hands.
We have great plans for the future of Posterous. Our team is stronger than ever, and we are laser focused on developing better ways for you to share your content online. Thank you for believing in us, thank you for trusting us, and thank you for sticking with us through the past six days.
This sort of crisis management is always a challenge. How do you communicate this sort of problem to customers without losing credibility later if there is a risk of lingering issues? Based on the tone of the email that you posted, as a customer, I would expect the problem solved, case closed. Certainly, it makes customers feel good that the team has it handled.
The new events, if not handled quickly, could create doubt in customer minds and the team, as the optimistic outlook may not hold. I hope they beat this soon.
This isn't like the ones who DDOS'ed MS out of their hatred for it or the ones who blackmailed & DDOS'ed a gambling website when they refused to pay up.
I severely doubt this line of thought.. but they didn't make any friends with their "switch from X to us" campaign, and it only takes one bruised ego with a lack of ethics to stir up trouble.
That said, if it's like most other DDOS attacks I've seen reported, the target is probably a user of the service with Posterous merely being unlucky enough to host a particular site someone doesn't like. I vaguely recall another platform got DDOSed recently due to some Israel vs Iran type cyberwar.
Weebly said (I think in their YC Founders at Work interview) that they get multiple DDoSes a day, aimed not at them but at sites they host. Now they have good enough systems in place to deal with it that they said they often just don't notice.
I guess when you host enough people's content, it's only a matter of time before someone wants to DoS something you're hosting.
Maybe, and take it with a grain of salt, some people got infuriated by their recent switch to posterous campaign. I know this may be a very remote option, but in my opinion it's very possible.
The botnet controller could be extorting websites directly... wire $5000 to my paypal account or your website will go down. Anyone from Posterous care to confirm/deny in this case?
It's not uncommon. There was an article a long time ago telling the story of a DDOS attack on a gambling website, and how a guy who was a philosophy major (iirc) figured out how to beat it, and then formed a company providing the same service. Forgot the url/title/etc, but it was good. Plenty more available with a search.
Our datacenter is experiencing heavy packet loss. We're on the line with Rackspace now.
I don't see where they stated it was a DDOS attack. Packet loss can occur due to a large number of different issues.
EDIT: They just clarified with the following: The DDoS attackers have returned and evolved their attack around our countermeasures. We expect to be back online ASAP w/ @gigenet antiddos
It's interesting that they're basically saying Rackspace's network couldn't hold up and doesn't have an effective solution for DDOS victims on their network.
Gige's DDOS protection is basically a hosted redirect and filtering system that "leverage[s] the cost of DDoS mitigation amongst a large group of businesses, giving you access to the infrastructure that would normally be out of reach financially." http://www.gigenet.com/ddos-protection.html
But if Posterous is hosted on the Rackspace cloud, why isn't there a mechanism to do this through their existing host? After all, Rackspace has an equivalent service (maybe rebranded?) that would seem to offer protection without needing to go to a 3rd party vendor. http://www.rackspace.com/managed_hosting/services/security/d...
So it's interesting that Posterous needed to go around Rackspace to get a solution.
It's tough -- if someone is hitting you at > 1gbps, that exceeds a gigabit ethernet port -- typically on shared services you only pay for that much. Any more than that and it affects all the other customers on the shared network. So yes, Rackspace has the bandwidth for it, but at the same time if you were Rackspace, would you let our DDoS take down the other hosts in the datacenter?
Gigenet / Black Lotus / Prolexic comes in and says -- we have excess bandwidth, we'll take care of it. Then they proxy clean traffic to you. Unfortunately as we are learning, it's also very expensive.
It wasn't classless, it was marketing. No one wants their competitors poaching their clients but in business you deal with it. That said, the same thought went through my mind, I wonder if it's related.
I could claim with nearly all certainty that this is a result of their campaign. The attacked several communities. Not just startups, but communities with developers (wordpress, tumblr etc). Their marketing was bold; but looking back they might have been more subtle.
Usually it means using a hardware solution like Intruguard or going through a third party vendor that resells Intruguard and/or other proprietary anti-DDOS technology.
We've opted to use Gigenet, and they're holding the line right now quite well.
TODAY, the attackers hopped IP's to our new IP immediately. So they appear to be learning. But then again, so are we.
Gigenet's anti-DDoS service has helped us a ton here and is now serving as our front door IP to block the synflood. They've been really responsive.