Dreamhost spam tools/virus scanning

apps

#1

Hi. The place where I work is considering using Dreamhost for email. I’ve already emailed support about an issue we need to be sure about.

In the meantime, I’ve read up on procmail as a spam solution. One of the comments in the version 2.0 of procmail in the knowledgebase was about procmail not working on m999999 style accounts, only full users. The vast majority of our users would be the m999999 style. I’ve also looked at filtering through the web panel, but I’d probably directly use procmail. Since we’d have many users, how about applying constantly changing rules to all my users? (Perl script to write a new procmail overtop the old one in each account? How does m99999 styles work in that regard?) I’m open to suggestions about how to approach this/flaws in my plans.

Also, I found a service that would scan email for virus’s and notify. But is there a virus scanner I could run to apply to all of a domain or sub-domain’s email accounts on the server?

Thanks.

Jim


#2

I’d use something like Spamassassin, rather than straight procmail filters. I can send you directions for this via email if you like, or check the archives of this forum.

Other than simply using regular users (rather than m999999 users) for all accounts, your best bet is probably to define a catchall, and then use perl / procmail / Spamassassin to redirect and filter mail. This could be tricky to implement properly, however, especially if messages are sent to multiple recipients at your domain. Basically, the ‘mailbox-only’ accounts are designed to be pretty much just that, without access to anything else.

You may also want to look into client-side spam / virus filtering, although this is, of course, undesirable for a number of reasons.

Last possibility (if you have the technical know-how, or if you are a masochist, and if you have a connection that’s usually or always online), would be to setup a FreeBSD or Linux box locally, and forward mail to that (or retrieve it, using something like getmail or fetchmail). Then you can setup spam filtering / virus scanning pretty easily; obviously this may not work at all for your particular situation.

We use RAV antivirus on our office mail server here at DreamHost HQ, and it works pretty well. The new version has a spam filter too, although I haven’t had time to play around with the new version yet. There is a (per-domain) licensing fee, but it’s fairly reasonably priced. There are a number of other similar solutions as well.


#3

Thanks Will. :slight_smile:

I’ll check out SpamAssassin in the archives and then post here if I need you to email me. I need to implement m999999 style mailboxes because I wont be able to afford the number of “users” that I can get in true POP3 mailboxes. Also, something I just noticed. I should have used spam filtering tools in my subject LOL.

I got a green light from Dreamhost support on my intentions; I’ll now give further details of my intentions. Initially, my boss wanted us to run a mail server in house. I used to work at an ISP though on the office management/dialup tech support management side. Former coworkers there said stay away from sendmail due to my lack of experience. Consultants suggested Exchange which is overkill; too costly also. To get up, going quick and easy, he’s now leaning toward Dreamhost.
All our boxes are Windows server boxes and I can’t deviate from that. :frowning: Also, any time I spend on this project just adds to the number of salaried hours I work; I prefer outsourcing for that reason alone. I’d love to learn this stuff, though. But this is just such a small part of what my duties are.

I work at a federally chartered credit union. Our webhost will continue to host our website (at least for now.) They presently point dns for a subdomain to our in-house secure site for internet banking. I intend to point their MX to Dreamhost for you guys to handle mail only. (It doesn’t make sense to me to only use you for mail, but that is what the boss wants to do for now.) Our current webhost would charge 1.75 per month per mailbox (for 100 email boxes…their lowest rate) or 2.75 per month per mailbox with server spam and virus filtering. Outrageous!!! $3 per mailbox per month (or $4 with virus/spam filtering) is their highest rate for 5 email boxes. I think they don’t want to do much email stuff. :slight_smile: They do have telephone support and Urchin webstats; but based on my experience I haven’t needed telephone support. I would miss Urchin; used to use Webtrends.

Support said this scheme will allow us to use mail addressed to jim@maindomain.org instead of having to address it to jim@mail.maindomain.org all the while continuing to address http traffic to our current webhost. Of course, we have current definitions of virus on every box. But I’d like to implement mail server virus scanning tho. A service we’d pay extra for would have our present host point MX to them and then once its scanned end up at Dreamhost. A workaround at least. My boss does want spam filtering at the server in place.

With more understanding of the hoops to jump through, any red flags or modification of suggestions?

Thanks.

Jim


#4

You might check out Mercury as far as Windows MTAs. I haven’t used it, but it’s supposed to be good.

Of course getting your company to deviate from the Windows only policy would be an even better idea… even if you’re using Exchange or something similar, setting up a UNIX / Postfix based SMTP gateway to deal with incoming mail, scan it, and then pass it along to the Exchange server is probably a good idea (to protect the Exchange server from the outside world).

The only problem with this is that Postfix can totally overwhelm an Exchange server, or so I’m told.

In any event, my guess is that sending all mail to a single mailbox and then doing some sort of processing from there is probably going to be your best bet; however it will be a bit tricky to get this to work perfectly, so you might look around for an existing script of some sort that will do what you want.


#5

I’d be a little wary of using Spam Assassin. Depending on how it’s set up, it can be disastrous to use in a business setting, because it tends to flag any references to money, business or profit as being signs that an e-mail is spam. Even a toll-free phone number pushes a message up the scale on their spam recognition system. Some people who’ve installed it have ended up having virtually all their e-mail flagged as spam.

There’s an article on it at http://www.talkbiz.com/assassin.html if you want to find out more.

Lynna

Business: http://www.spidersilk.net
Personal: http://www.wildideas.net


#6

Note that the person writing this article apparently has no understanding of how SpamAssassin actually works. Many of the things that SA scores on may seem somewhat arbitrary, but the scores are based on actual spam and non-spam messages and tend to do a pretty good job of predicting what’s spam and what isn’t. Additionally, many of the scores the guy in that article mentions have a low positive, or even a negative score.

A SA user can set their own score, and can score certain tests higher or lower. For instance, if a particular score is causing a lot of messages to be flagged as spam because of one test, you could reduce the weight of that score.

As with any spam filtering / blocking software, you should always go through the messages that were blocked to doublecheck that you didn’t miss any important mail.

I’ve used SA for quite some time, and the number of false positives / negatives is really low (though, granted, I filter most of my incoming mail from mailing lists and such before spam filtering).

The guy who wrote that article appears to be a total idiot, with a very poor understanding of how email (and SpamAssassin) work. My guess is that he’s just sore that a message he sent was blocked by SA.


#7

[quote]I’d be a little wary of using Spam Assassin. Depending on
how it’s set up, it can be disastrous to use in a business
setting, because it tends to flag any references to money,
business or profit as being signs that an e-mail is spam.

[/quote]

It depends on how you view SpamAssassin. I highly recommend that you don’t automatically delete anything caught by it, but rather use your client-side filters to move any messages it marks into a “Potential Spam” folder. Then, at the end of the day, quickly scan through the list of messages and pull anything out that looks legitimate.

This way, your daily routine isn’t interrupted by throwing spam out manually every time it ends up in your inbox, and you don’t have to “context shift” into spam-fighting mode dozens of times a day. If something legitimate gets filtered away - which happens very rarely for me - the worst that happens is that it takes a few hours more for you to send your reply.

It also takes a little effort to tweak your SpamAssassin config to fit your particular email profile. If you receive lots of stuff dealing with money/profit/etc. then you should lower the weights for those particular tests. Make sure to add known addresses to your white-list, and so on. It takes a couple of hours to get everything just right, but if you get any amount of email at all this will save you tons of time in the end.

(If anyone has a Mac, I also highly recommend getting Mac OS X and using the bundle Mail program’s spam filter - in my experience it works nearly as well as SpamAssassin…)

  • Jeff @ DreamHost
  • DH Discussion Forum Admin

#8

[quote]Note that the person writing this article apparently has no
understanding of how SpamAssassin actually works. Many of the
things that SA scores on may seem somewhat arbitrary, but the
scores are based on actual spam and non-spam messages and
tend to do a pretty good job of predicting what’s spam and what
isn’t.

[/quote]

I think that would depend on what kind of mail you receive. Many of the keywords listed, while certainly popular in spam, can quite legitimately turn up in non-spam mail as well. And if someone’s running a business, or particularly if they’re publishing a business newsletter like the author of the article does, then they’ll turn up more often. Things that might not be an issue for one user might be for another – which is fine if everyone has their own individual setup, but some ISPs install this at the server level and don’t let the users control it.

[quote]A SA user can set their own score, and can score certain tests
higher or lower.

[/quote]

If they have their own installation of it, and if they know how. If their ISP installs it, especially without their knowledge, then they don’t have that option. And that does happen… My wariness of Spam Assassin isn’t only based on this article, but on horror stories I’ve heard from other people who’ve had problems with it.

[quote]As with any spam filtering / blocking software, you should always
go through the messages that were blocked to doublecheck that
you didn’t miss any important mail.

[/quote]

If you can. If your ISP installs it, and particularly if they set it up to delete suspected spam, then you can’t.

[quote]The guy who wrote that article appears to be a total idiot, with a
very poor understanding of how email (and SpamAssassin) work.
My guess is that he’s just sore that a message he sent was
blocked by SA.

[/quote]

I don’t think a personal attack on someone who’s not here to defend himself is very helpful or appropriate. I don’t know if you actually read it all the way through, but the article refers to problems a number of different people have had with Spam Assassin and similar programs, plus the program’s documentation and a talk with one of the developers, so it’s certainly not based on any single incident. If you have specific technical points to make, that’s great and I’d love to hear them, but calling people names doesn’t really strengthen your argument or prove anything.

The whole spam filtering issue is such a messy one. I get deluged with spam on a daily basis and what filtering I’ve been able to set up myself with the filters in the web panel helps some, but not enough. I still have to deal with a lot of it manually. I’d love to find a filtering system that actually works, but a lot of the one’s I’ve seen just seem like recipes for trouble, whether they’re based on content or origin.

At one point, a major ISP in the US that a lot of my contacts there use was blocking any mail I tried to send to any of their clients because they’d had a report of someone using my ISP to send spam. I use the largest ISP, and at the time the only DSL provider, in Canada – they’re used by something like 50% of the Canadian population, so in response to one spam incident the American ISP had blocked half a country! I’ve heard of other blacklists that have excluded pretty much the entire continent of Asia – blocking all domains with Asian country codes because spammers often use Asian ISPs. What does that do to people who have friends, family or business contacts there?

Another time one of my clients sent me (along with presumably all their other contacts) a list of domains they recommended people block as likely sources of spam. It included every major webmail provider, several large national or international ISPs, a couple of large computer manufacturers, and a handful of domain registrars, including the one that one of their own domain names was registered with! Plus all Asian country codes as mentioned above. Anyone who implemented that list would lose a ton of mail, including some very important things like domain renewal notices. I wrote back to them pointing that out, and asked why some of these companies were included on it, but just received a terse reply to the effect that the sources of the list were confidential, but that people could modify it if they wanted to. But I’m sure many people who were less critical probably did implement it unquestioningly, and lost piles of mail as a result.

It’s just getting to the point where I’m not sure which is more of a headache, spam or spam filters… :frowning:

Lynna

Business: http://www.spidersilk.net
Personal: http://www.wildideas.net


#9

(I’m speaking for myself and not DreamHost, these opinions are entirely my own, etc. etc. etc.) :>

Hi Lynna -

[quote]If you have specific technical points to make, that’s
great and I’d love to hear them, but calling people names
doesn’t really strengthen your argument or prove anything.

[/quote]

Agreed. Will’s opinions are his own, as are mine (well, not his own, but my own). In any case, while I wouldn’t have characterized it the same way - and hope he is more diplomatic about these sorts of things in the future - Will is totally right in that the article’s author doesn’t really seem to present a viable argument against filtering technologies.

He also seems to be insulting those who recommend SpamAssassin, so I’m not what I should think - both myself and Will would be included in that group, as would other people in this forum:

" In my opinion, any person that recommends the use of any
" content-based email filtering system should be required to
" re-take their IQ test, to see if they still have one.

The sad fact is, legitimate email CAN be filtered incorrectly. It happens to me occasionally, too. Spam filters are getting better and better all the time, though, and in my opinion the ultimate responsibility for any trouble they cause should rest entirely on the shoulders of the spammers whose incessant need for ‘market penetration’ has necessitated them.

[quote]If they have their own installation of it, and if they
know how. If their ISP installs it, especially without
their knowledge, then they don’t have that option. And
that does happen…

[/quote]

If an ISP installs something like SpamAssassin and doesn’t tell its users - and/or doesn’t allow them to change their weights or whitelist people - customers will leave. And rightly so.

This isn’t something that SpamAssassin should be blamed for, but rather those ISPs that have so little regard for their users that they don’t involve them in the process. Any sufficiently useful technology can be used in boneheaded ways to the detriment of those who depend on it. This applies to DreamHost as much as anyone else, and I’d hope that if we were to do something that insane you guys would keep us in check, starting with angry emails and progressing from there. Having been here for a while, I’d be kind of surprised and disappointed if you didn’t. ;>

While it’s harmful from a business perspective for a certain percentage of your customers to not hear from you due to policy foul-ups, in the vast majority of cases people configure and fine-tune their filters themselves or otherwise chose to use the filters with the knowledge that it could block legitimate email in mind. Assuming that their web host or ISP made it clear that this could happen, any blame for missed email falls on them as individuals. They’d be hurting themselves, and will probably learn from their mistakes given some time.

[quote]At one point, a major ISP in the US that a lot of my
contacts there use was blocking any mail I tried to send
to any of their clients because they’d had a report of
someone using my ISP to send spam.

[/quote]

I don’t know about this specific case, but again it depends on the blacklist. Generally speaking the only time someone in blacklisted is when they (as a service provider) have received numerous complaints and have neglected to do anything about it. While the fact that your provider had a de-facto monopoly on providing DSL in your location surely complicated things, I can only presume that they cleaned up their act once their customers started complaining.

It’s good to remember that blacklists are somewhat different than ‘spam filtering’, the latter of which tends to be configurable on an individual level. I have absolutely no problems with spam filtering, and in most cases blacklists are run fairly. While there are some cases where I think they are a bit draconian, but nobody ever forces an ISP to use a blacklist so it’s their call.

I should remind you that both Will and I are coming at this from a somewhat unique perspective. Spam isn’t just an annoyance to us, but something that can kill our business. It costs us money and being listed in a blacklist could easily be a kiss of death for a web host - and yet, we still support them (in most cases). Spam in general hurts everyone no matter how good on spam they are, especially service providers like DreamHost (and ultimately, it hurts our customers too - even if only in the higher prices we have to charge to make up for it).

You could say that I don’t have a lot of sympathy for “spam-friendly” hosts. It’s an ethical choice any business makes, and if they get bit for making the wrong one I hope their users take them to task for it.

For the record, DreamHost HAS been blacklisted before, due to the actions of one of our upstream providers (we’re usually really good about keeping our own house clean). Without going into tons of detail they were very lax on their policies, and some of us here even think they had ‘pink’ contracts with known spammers. Due to this and other reasons we decided to leave, and last I checked they had gone bankrupt. While we and some of our customers were hurt to some degree by being included in the blacklist, ultimately that economic pressure found the source of the spam and solved the root problem. We are blocked no more.

Another thing: Unlike Will, I am not only involved in handling spam issues but am also in the DreamHost Marketing Team (which is actually my main purpose here). I understand first-hand what sorts of pressures small business operators have to deal with, and how hard it can be to promote yourself. However, I also know that email is an incredibly valuable business resource and if it continues to be ‘polluted’ by spam then people will inherently distrust anything commercial in nature they receive by email - or stop using it entirely. That’s a much bigger threat to online commerce than spam filters and blacklists, IMHO.

  • Jeff @ DreamHost
  • DH Discussion Forum Admin

#10

I don’t think a personal attack on someone who’s not here to
defend himself is very helpful or appropriate. I don’t know if you
actually read it all the way through, but the article refers to
problems a number of different people have had with Spam
Assassin and similar programs, plus the program’s
documentation and a talk with one of the developers, so it’s
certainly not based on any single incident.

[/quote]

I haven’t read the article recently (it makes me too mad, for one thing), but I read it a couple of times a while back (mainly when you linked it in a kbase article, which I also responded to).

He may have read the documentation, but I don’t think he really understands how the scores actually work, or why they’re effective.

He’s also complaining about some peoples’ specific implementation of SA - as used to actually reject incoming mail. I don’t think anyone here is advocating that. I don’t /dev/null any mail marked by SA (although anything with a score over 15 or 20 is almost definitely spam).

If he wanted to get mad at a particular ISPs use of the tool and / or their configuration, that’s one thing, but blame the appropriate people here. SpamAssassin is a tool that can be helpful to some people when configured / used properly. As with any piece of software, it can be configured improperly, or used for things it’s not really designed to do, but you can’t really blame the authors of the software for that.

His objections are mostly to tests that have low positive or even negative scores, and the scores (again) are based on actual spam, and the liklihood that they contain a certain element. Whether or not you agree with the tests on paper, they’re very effective.

Quotes:

[quote]"Talks about bulk email: I LOVE this one. Every email that
discusses STOPPING spam is subject to various filters in this
guy’s list. Including a lot of spam complaints to ISPs that use
Spam Assassin…

Hey! Spammers! Want a safe haven? Sign up for accounts at
ISPs that use this software! They’ll probably never see the
complaints!"

[/quote]

ISPs almost never filter abuse@ or postmaster@ accounts for obvious reasons. For instance, we use some internal blocklists, but postmaster@, abuse@, etc. aren’t blocked; neither is my personal address as it’s also an ARIN contact (which occasionally receives spam complaints and the like). I don’t use any content filters at all on the abuse mailbox either… obviously that would be stupid, as one expects to receive spam at this address.

Now if a spammer does send spam to our abuse address, I am almost certain to report it, and very possible to add the originating IP address to our internal blocklists. Spamming abuse@ or postmaster@ is a really stupid thing to do.

We do use SA a little bit on our support box, although the scores have been heavily modified and mail from any email address that’s in our customer database is accepted regardless of the score. That’s because the amount of spam (especially spam with foreign character sets) was making it very difficult to get work done.

[quote]“Talks about opting in: Ah, what the hell. No publisher ever uses
that phrase, right? And online business publications that
espouse proper marketing practices never admonish you to stick
to opt-in vs spam, do they?”

[/quote]

I don’t know about you, but I’ve rarely / never received an email talking about opting in that wasn’t spam.

[quote]“Of course, the programmers of this type of software suggest
that the software be used in “conservative” ways. This is
somewhat akin to saying that you should shoot people gently.”

[/quote]

I dont’ think this analogy really makes sense; I think that the authors of the software are sincere in wanting it to be used in a way that’s appropriate to the application.

Is SA appropriate for naïve users? Probably not, which is one reason we haven’t made instructions for setting it up / using it widely available. Anyone using any type of spam filtering system should be responsible for using it properly and checking caught messages once in a while.

If you can. If your ISP installs it, and particularly if they set it up > to delete suspected spam, then you can’t.

[/quote]

I don’t think anyone here is advocating this type of behavior. No filter which blocks based on content is 100% effective, and I don’t think many responsible ISPs use such a filter to completely prevent mail from reaching a customer.

If we were ever to implement server-side content filtering, the strongest action we’d take would probably be to tag messages with some sort of score.

[quote]It’s just getting to the point where I’m not sure which is more of
a headache, spam or spam filters…

[/quote]

For me, I’m 100% sure of what the answer is… Without spam filters, I’d go completely insane. I’ve received over 1500 pieces of spam to my main email account sine Jul 31, and lots more at other accounts. Having this type of mail end up in my inbox totally messes up my organization scheme. I go through my spam folder 3 or 4 times a day and doublecheck that there aren’t any false positives in there.

I would like to check out some of the newer pseudo-bayesian spam filters when I have some spare time.


#11

OK, that’s more the sort of detailed response I was looking for. It does sound like maybe a lot of the problems I’ve heard about have been due to people setting it up stupidly.

I’d be half-tempted to install it myself and try it out (without deleting anything, obviously!), but my home system isn’t Linux, it’s a Mac, and I can’t tell from the docs whether there’d be some way to set it up in my DH home directory or whether it would need to be on my own system. There do seem to be a handful of plug-ins for using it with various popular e-mail programs, including Eudora which I use, but they’re all for Windows. I tell you, no one likes Mac users. :-/

Might try the OS X mail program that Jeff mentioned once I get a machine that can actually handle OS X. Right now I’m still using an ancient beige G3 and can’t afford to upgrade to anything newer for a while, since this has not been exactly a banner year for the web industry in general or my own web design business in particular. :frowning:

In the meantime, I guess I’m stuck with setting up my own procmail filters, which is better than nothing but not by a whole lot. I get a LOT of spam – about as much as you do, from the sounds of it.

Oh, and BTW – yes, I can think of mail I’ve gotten that mentioned opting in that wasn’t spam. Two Dreamhost newsletters and four announcements, all related to the confirmed opt-in policy for mailing lists you guys brought in last February. So there! :slight_smile:

Lynna

Business: http://www.spidersilk.net
Personal: http://www.wildideas.net


#12

It’s installed on the mail servers… I have instructions for setting it up somewhere if you hit me up via email or through the support system.

You can also install your own copy in your home directory; there’s some information on doing this in the SA documentation.


#13

PS - I’ve been checking out bogofilter. It’s interesting so far, but I’ve been getting a lot of spam to my inbox still. We’ll see how it goes after some more “training”.

Has anyone used any of the other bayesian / pseudo-bayesian filters?