In addition to the good advice you have already received, I would only add that you not just ignore it. I did, once, after allowing a "well known and widely used PHP form-maililng script" that I did not personally inspect (closely enough) remain in a site I was migrating from another host for a client.
I started seeing what you are seeing and said to myself, "I really ought to look into that!" but I didn't do it right away. After a few flurries of these emails, usually arriving in "bunches" once or twice a day, and then nothing, I figured it wasn't that big an issue, and kept shoving it on the "back burner", prefering to spend my time on "paying" work. Big Mistake.
The next thing I know, the spammers/script-kiddie/low-lifes had succeeded in putting together a string that "worked", and I received an email from DreamHost that my user had been restrcted from sending email due to apparant spam activity and over 200 emails per hour.
It took a rather frenzied dive into my stats and logs to identify the culprit as, at that time, many of my sites were running as the same user. Before I found the problem, exorcised the exploited script, and convinced DH that I was not a spammer (BTW, I think longevity with your hosting provider helps with that! Probably something those who switch hosts at the first sign of trouble should consider). several hours had gone by during which none of the sites running as the affected user were able to send mail. Bad Webmaster. Disgruntled Clients. General Very Bad Thing.
Final note: I have never had the version of formmail.cgi provided by Dreamhost exploited. Use that one, especially if you are not sure how safe the one you are using, or have coded, really is.
Sharing the three lessons I learned from this:
1) Don't trust any script you don't thoroughly know, and understand. If you really can't, or won't, inspect the thing yourself and know it is sufficiently "hardened", trust only those scripts that are vouched for by those who you know can and will. "Popularity" or "common and widespread use on the net" are not an acceptable substitute for this process, and niether is the fact that a script cost you a lot of money from a "professional" or a "script selling website." In fact, you can argue that the very popularity and wide usage of a script makes it more likely to be exploited because attacking it sucessfully will open so many doors (think IE exploits here).
2) Never ignore indications of activity that might be "hackish" or "crackish" in nature or intent. Just because it didn't appear to work for the perpetrator does not mean they won't keep trying. If they do keep trying, they must believe they will succeed (they know what script you are using, or know the flaw they are trying to exploit), and if that is true, and you don't do something to prevent it, they probably will succeed.
3) Don't host multiple sites under the same user name. I know it is by far the easiest way to manage things at Dreamhost, but take the extra step and segregate sites by user. This can minimize your exposure if an exploit occurs by only having one "problem user" to deal with and make it much easier to locate the offending script, visitor, bot, etc.