"weblogcomment spam causing server instability"?


#1

Could anyone please explain to me how is it possible that specifically “the increase in comment spam has
been causing a lot of server instability” as said by DH below? Or is it actually simply that the increase in comments has been causing the instability?

Chris

Weblog Comment Spam (Downtime)

Posted: Dec 15th, 2004 - 04:06:31 PM PST (20 days 16 hours ago)

We have seen a significant increase in weblog comment spam lately. Movable Type installations
seem to be the worst hit, but Greymatter is also affected. The increase in comment spam has
been causing a lot of server instability. We have begun blocking connections from the IP
addresses we have found to be the origins of most of the spam


#2

It’s directed to comment spam specifically, not legitimate comments. Why? Because legitimate commentators don’t post 500 comments in the span of 2 seconds.


MacManX.com


#3

You’re suggesting that 500 comments in 2s causes instability?


#4

500 spam comments in 2 seconds on over 10,000 blogs (there’s way more than that on the DH servers), accounts for at least 150,000,000 database and/or PHP queries per minute, or 2,500,000 per second. I’d say that would cause some instability, wouldn’t you?


MacManX.com


#5

Load, not instability. I’d hope DH would protect the stability. And if that means limiting the load, do that, rather than spending effort blocking whatever IP addresses happen to be causing the load today. And which might be reassigned to a legitimate user tomorrow.


#6

Overloading a server often causes instability. This is why Distributed Denial of Service (DDOS) attacks are so popular with hackers. The server becomes so busy handling illegitimate requests that it can’t handle legitimate requests fast enough.

As the legitimate requests begin to timeout, those requests are also retried. The server also has to context switch between handling network requests, I/O requests, running system processes, running user apps, etc. The situation continues to snowball until the server appears completely unresponsive, or even crashes.

The problem is that the servers need to be able to distinguish between legitimate and illegitimate requests, and they need to do this really, really quickly. DreamHost doesn’t have an easy way to do this in general, since that would require that they make a lot of assumptions about the programs that each one of us are running on the shared servers. All combined, users are running thousands of unique applications on these servers, and new vulnerabilities are reported each day.

Over time, DreamHost can detect common issues like comment spamming, and then they can start setting up blacklists, etc., to block the incoming requests. This isn’t easy to do, though, without causing side effects that legitimate users are affected by. A more effective solution is to handle as much of the problem as possible at the application. If you keep up to date on patches and make it hard for hackers and spammers to even attack your apps, then DreamHost will have a better chance of thwarting the smaller number of them that do gain a foothold.

Limiting load on a single server is also a challenging problem. How do you choose what to limit? The easiest approach is to move some of the processes (which typically implies users) to another server. That’s not quick, though, and it’s really hard to do when a server is already under attack. Linux and Unix do have some effective ways to adjust the priorities of processes, but again, it’s hard to do when you’re already under attack. It is also very time consuming for the system administrators to deal with on a case by case basis.

I’m part of an engineering team that runs applications we have written on Linux-based app server clusters that are co-located at multiple data centers, and I can speak from personal experience as to how challenging of a task it is to keep servers secure, yet doing their jobs, with minimal to no downtime.

We should all be glad we aren’t on Windows servers (of course, many of us wouldn’t be here if DH used Windows). I’ve had to write, deploy, and manage apps on Windows servers at previous jobs, and it feels like you are fighting hand-to-hand combat with one hand tied between your back and sand thrown in your eyes. Attempting to control process priority on a Windows server is generally an exercise in futility. It’s easier on Linux, but it’s still not a walk in the park.

Robert
http://www.wombatnation.com/


#7

[quote]Overloading a server often causes instability.
Limiting load on a single server is also a challenging problem. How do you choose what to limit?

[/quote]

For whatever your definition of “overloading” is (something which I’m not clear on, even having read your explanation), you choose to limit the work so to avoid “overload”.

[quote]The problem is that the servers need to be able to distinguish
between legitimate and illegitimate requests

[/quote]

Actually I think the problem is first that the /provider/ needs to distinguish. E.g. within the Terms of Use. One cannot resonably complain about “illegitimate” use without an objective definition, and a definition based solely on whether the use causes overload is wholly unsafe.


#8

Load does have several different meanings, with no pun intended about overloading the meaning of overloaded :). It’s sometimes hard to clearly distinguish between CPU load, network I/O load, memory access load, disk I/O load, etc., because they end up being fairly interconnected. You still keep coming back to the decision of prioritizing exactly which processes to limit. There are a lot of different processes running on the shared servers. What may be a reasonable number of inbound comments on one blog might be hundreds of times normal on others. Understanding what is normal behavior for these processes is difficult, although DH already puts controls (e.g., limits on the length of time a process can run) on user processes to minimize the impact they can have on other users.

Of course another vague word is “instability”. Sometimes instability is used to describe situations where server response time is varying widely. To a user, this behavior could be interpreted as instability, especially if their web browser times out while waiting on a response from a web server.

Although a crude tool, blacklisting IP addresses can work well if used carefully. Obviously, you don’t want to blindly block IP addresses that are being dynamically handed out by ISPs via DHCP to home users. However there are ranges of dedicated IP addresses that are well known to be used only by spammers. Blacklisting needs to be just one of many tools used to manage servers exposed to the Internet.

Robert
http://www.wombatnation.com/


#9

[quote]DH already puts controls (e.g., limits on the length of time a process can run) on user processes

[/quote]

Interesting… I’ve not seen a limit on e.g. PHP scripts (I’ve had them run for e.g. 10 mins) and find no mention in KB. Can you refer me to any docs for these limits?

[quote]to minimize the impact they can have on other users.

[/quote]

I’d be disapointed with a provider that aimed to minimise that impact. I think user of a shared server expect and accept impact.

[quote]Blacklisting needs to be just one of many tools

[/quote]

Agreed. But in this case of blog comment, seems it is not.


#10

Here’s a kbase article on persistent processes:
https://panel.dreamhost.com/kbase/index.cgi?area=2449

It doesn’t specifically say anything about killing long running processes, but this used to be an issue for people with large Movable Type blogs. With older versions of MT if you wanted to rebuild your site with a new layout, you needed to run a script that could take many minutes to run. For people with really large blogs, it could take more than thirty minutes.

DH was running, and may still be running a high priority process that monitored long running processes and killed them. I’m not sure what criteria was used, i.e., how long, only processes that behaved like they were in an endless loop, etc. Over time, DH made some changes to minimize or eliminate this problem with respect to MT for most people, if not everyone.

Robert
http://www.wombatnation.com/


#11

[quote]Here’s a kbase article on persistent processes:
https://panel.dreamhost.com/kbase/index.cgi?area=2449

[/quote]

Interesting to see “We define persistent/background processes as any unix user’s command running non-interactively” excludes PHP scripts lauched by CGI. Unless the including Apache itself! :wink:

[quote]DH was running, and may still be running a high priority process that
monitored long running processes and killed them.

[/quote]

Disappointing, since long-running doesn’t necessarily have any undue adverse effect. Mind you, since that article’s stated disapproval is for processes that is “in any way adversely affecting the smooth functioning of your shared server” which taken at its word includes all processes, it seems DH have not really thought this through properly.