SpamAssassin broken since Sarge upgrade

apps

#1

I’m hoping there are some SpamAssassin (SA) experts who can help. This all started about the time DH upgraded to Sarge. I first noticed the problem when my cron job failed to update my SA Bayes database. I finally tracked the problem to being an outdated Berkeley database format. However, once updated, SA could no longer gain access or “tie” to the database for using Bayes tests on incoming mail. So, I upgraded SA.

After removing the old 3.0.2 version and installing 3.0.4, I recreated the Bayes databases (sa-learn) from my e-mails; no import or upgrade to any previous databases. However, I still have the same problem. Here’s the error as reported in my procmail log file:

Cannot open bayes databases /home/ccrumley/.spamassassin/bayes_* R/O: tie failed:
Cannot open bayes databases /home/ccrumley/.spamassassin/bayes_* R/O: tie failed:
Cannot open bayes databases /home/ccrumley/.spamassassin/bayes_* R/O: tie failed:

It’s failing to “tie” to the bayes_seen, bayes_toks, and auto-whitelist databases. I’ve tried the db_dump/load tip as listed here but with no success. It could possibly be a problem with Perl’s DB_File but I wouldn’t know how to fix that.

Any ideas? I’m not sure what version of SA that DH is running, but if they keep it current, I might consider abandoning my self-installed copy.

-Matt


Here’s the output from running “spamassassin –D –lint”:

debug: SpamAssassin version 3.0.4
debug: Score set 0 chosen.
debug: running in taint mode? yes
debug: Running in taint mode, removing unsafe env vars, and resetting PATH
debug: PATH included ‘/usr/local/bin’, keeping.
debug: PATH included ‘/usr/bin’, keeping.
debug: PATH included ‘/bin’, keeping.
debug: PATH included ‘/usr/bin/X11’, keeping.
debug: PATH included ‘/usr/games’, keeping.
debug: Final PATH set to: /usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
debug: diag: module installed: DBI, version 1.46
debug: diag: module installed: DB_File, version 1.808
debug: diag: module installed: Digest::SHA1, version 2.10
debug: diag: module installed: IO::Socket::UNIX, version 1.21
debug: diag: module installed: MIME::Base64, version 3.04
debug: diag: module installed: Net::DNS, version 0.48
debug: diag: module installed: Net::LDAP, version 0.32
debug: diag: module not installed: Razor2::Client::Agent (‘require’ failed)
debug: diag: module installed: Storable, version 2.12
debug: diag: module installed: URI, version 1.35
debug: ignore: using a test message to lint rules
debug: using “/home/ccrumley/saetc/mail/spamassassin/init.pre” for site rules init.pre
debug: config: read file /home/ccrumley/saetc/mail/spamassassin/init.pre
debug: using “/home/ccrumley/sausr/share/spamassassin” for default rules dir
debug: config: read file /home/ccrumley/sausr/share/spamassassin/10_misc.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_anti_ratware.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_body_tests.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_compensate.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_dnsbl_tests.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_drugs.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_fake_helo_tests.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_head_tests.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_html_tests.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_meta_tests.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_phrases.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_porn.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_ratware.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/20_uri_tests.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/23_bayes.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/25_body_tests_es.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/25_hashcash.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/25_spf.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/25_uribl.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/30_text_de.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/30_text_fr.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/30_text_nl.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/30_text_pl.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/50_scores.cf
debug: config: read file /home/ccrumley/sausr/share/spamassassin/60_whitelist.cf
debug: using “/home/ccrumley/saetc/mail/spamassassin” for site rules dir
debug: config: read file /home/ccrumley/saetc/mail/spamassassin/local.cf
debug: using “/home/ccrumley/.spamassassin” for user state dir
debug: using “/home/ccrumley/.spamassassin/user_prefs” for user prefs file
debug: config: read file /home/ccrumley/.spamassassin/user_prefs
debug: plugin: loading Mail::SpamAssassin::Plugin::URIDNSBL from @INC
debug: plugin: registered Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x858a520)
debug: plugin: loading Mail::SpamAssassin::Plugin::Hashcash from @INC
debug: plugin: registered Mail::SpamAssassin::Plugin::Hashcash=HASH(0x8ed806c)
debug: plugin: loading Mail::SpamAssassin::Plugin::SPF from @INC
debug: plugin: registered Mail::SpamAssassin::Plugin::SPF=HASH(0x8ea5f68)
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x858a520) implements 'parse_config’
debug: plugin: Mail::SpamAssassin::Plugin::Hashcash=HASH(0x8ed806c) implements 'parse_config’
debug: bayes: 1311 tie-ing to DB file R/O /home/ccrumley/.spamassassin/bayes_toks
debug: bayes: 1311 tie-ing to DB file R/O /home/ccrumley/.spamassassin/bayes_seen
debug: bayes: found bayes db version 3
debug: Score set 3 chosen.
debug: ---- MIME PARSER START ----
debug: main message type: text/plain
debug: parsing normal part
debug: added part, type: text/plain
debug: ---- MIME PARSER END ----
debug: metadata: X-Spam-Relays-Trusted:
debug: metadata: X-Spam-Relays-Untrusted:
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x858a520) implements 'parsed_metadata’
debug: dns_available set to yes in config file, skipping test
debug: decoding: no encoding detected
debug: URIDNSBL: domains to query:
debug: is Net::DNS::Resolver available? yes
debug: Net::DNS version: 0.48
debug: all 'From’ addrs: ignore@compiling.spamassassin.taint.org
debug: Running tests for priority: 0
debug: running header regexp tests; score so far=0
debug: registering glue method for check_hashcash_double_spend (Mail::SpamAssassin::Plugin::Hashcash=HASH(0x8ed806c))
debug: registering glue method for check_for_spf_helo_pass (Mail::SpamAssassin::Plugin::SPF=HASH(0x8ea5f68))
debug: SPF: message was delivered entirely via trusted relays, not required
debug: registering glue method for check_hashcash_value (Mail::SpamAssassin::Plugin::Hashcash=HASH(0x8ed806c))
debug: all 'To’ addrs:
debug: registering glue method for check_for_spf_softfail (Mail::SpamAssassin::Plugin::SPF=HASH(0x8ea5f68))
debug: SPF: message was delivered entirely via trusted relays, not required
debug: registering glue method for check_for_spf_pass (Mail::SpamAssassin::Plugin::SPF=HASH(0x8ea5f68))
debug: registering glue method for check_for_spf_helo_softfail (Mail::SpamAssassin::Plugin::SPF=HASH(0x8ea5f68))
debug: registering glue method for check_for_spf_fail (Mail::SpamAssassin::Plugin::SPF=HASH(0x8ea5f68))
debug: registering glue method for check_for_spf_helo_fail (Mail::SpamAssassin::Plugin::SPF=HASH(0x8ea5f68))
debug: running body-text per-line regexp tests; score so far=0.126
debug: running uri tests; score so far=0.126
debug: bayes corpus size: nspam = 223, nham = 206
debug: tokenize: header tokens for F = "Uignore D
compiling.spamassassin.taint.org D
spamassassin.taint.org Dtaint.org Dorg"
debug: tokenize: header tokens for *m = " 1126385256 lint_rules "
debug: tokenize: header tokens for *RT = " "
debug: tokenize: header tokens for *RU = " "
debug: bayes token ‘somewhat’ => 0.0489090909090909
debug: bayes: score = 0.264547785547786
debug: bayes: 1311 untie-ing
debug: bayes: 1311 untie-ing db_toks
debug: bayes: 1311 untie-ing db_seen
debug: registering glue method for check_uridnsbl (Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x858a520))
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x858a520) implements 'check_tick’
debug: running raw-body-text per-line regexp tests; score so far=-0.97
debug: running full-text regexp tests; score so far=-0.97
debug: Current PATH is: /usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
debug: Pyzor is not available: pyzor not found
debug: DCCifd is not available: no r/w dccifd socket found.
debug: DCC is not available: no executable dccproc found.
debug: Running tests for priority: 500
debug: RBL: success for 1 of 1 queries
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x858a520) implements 'check_post_dnsbl’
debug: running meta tests; score so far=-0.97
debug: running header regexp tests; score so far=0.256
debug: running body-text per-line regexp tests; score so far=0.256
debug: running uri tests; score so far=0.256
debug: running raw-body-text per-line regexp tests; score so far=0.256
debug: running full-text regexp tests; score so far=0.256
debug: Running tests for priority: 1000
debug: running meta tests; score so far=0.256
debug: running header regexp tests; score so far=0.256
debug: using “/home/ccrumley/.spamassassin” for user state dir
debug: lock: 1311 created /home/ccrumley/.spamassassin/auto-whitelist.lock.balrog.dreamhost.com.1311
debug: lock: 1311 trying to get lock on /home/ccrumley/.spamassassin/auto-whitelist with 0 retries
debug: lock: 1311 link to /home/ccrumley/.spamassassin/auto-whitelist.lock: link ok
debug: Tie-ing to DB file R/W in /home/ccrumley/.spamassassin/auto-whitelist
debug: auto-whitelist (db-based): ignore@compiling.spamassassin.taint.org|ip=none scores 0/0
debug: AWL active, pre-score: 0.256, autolearn score: 0.256, mean: undef, IP: undef
debug: DB addr list: untie-ing and unlocking.
debug: DB addr list: file locked, breaking lock.
debug: unlock: 1311 unlink /home/ccrumley/.spamassassin/auto-whitelist.lock
debug: Post AWL score: 0.256
debug: running body-text per-line regexp tests; score so far=0.256
debug: running uri tests; score so far=0.256
debug: running raw-body-text per-line regexp tests; score so far=0.256
debug: running full-text regexp tests; score so far=0.256
debug: is spam? score=0.256 required=3.5
debug: tests=BAYES_40,MISSING_HEADERS,MISSING_SUBJECT,NO_REAL_NAME
debug: subtests=__HAS_MSGID,__MSGID_OK_DIGITS,__MSGID_OK_HOST,__SANE_MSGID,__UNUSABLE_MSGID


#2

Okay, this is getting more confusing. You say DH is running SpamAssassin version 2.2? It must be a different machine than mine. I run:

/usr/bin/spamassassin -V

and it returns:

SpamAssassin version 3.0.3
running on Perl version 5.8.4


#3

I don’t think it’s SA; its the version of libdb being used by Perl. The Sarge machines are using a later version of Perl that uses a later version of libdb. (I’m using a self-installed version of SA 3.0.4.)

It turns out that libdb3 is installed on the Sarge machines. I downloaded the source to the DB_File Perl module and rebuilt it to use libdb3, but I couldn’t figure out how to get SA to use the modified DB_File. Perl isn’t my forte, so I probably did something wrong.


#4

Did you build SA with your own installed version of Perl? I’d be willing to try that option. I’m still not sure if DH is done with the upgrades, so I want to wait for the dust to settle and try installing SA later.


#5

I didn’t go that far, but that probably would work.

There was a note in another thread about SpamAssassin that DH would be upgrading its mail servers to Sarge in a few weeks and the problem would go away.


#6

I hope that’s the case. DB_File must be the problem. I upgraded to SA v3.1.0 and it’s giving me the same R/O tie failed problem when it tries to open the Bayes database files.

-Matt


#7

Can someone explain what runs on the mail server, and what runs on the “local” server where I have put my spamassassin code?


#8

Here’s my guess. (I don’t know how things really work.) When a message is delivered, all of the processing is done on the mail server. The mail server will, however, run your locally installed version of spamassassin if that’s how you set things up. It does this using its own version of Perl, which uses the older libdb. If you run sa-learn (manually or in a cron job) the code is running on the shared hosting machine using a later version of Perl, which uses a newer libdb.


#9

Interesting.
That could explain some of the messages I see when I set procmail to verbose logging (such as version of DNS is too low).
If I compile Razor, would razor need to be on the mail server as well?


#10

I’ve been reading SA documentation on how to use SQL databases for storing Bayes tokens. I don’t know if it will work but it’s worth trying. If I find that it works, I’ll post the steps here.

-Matt


#11

Good news!

This evening I decided to try the Bayes DB implementation using MySQL. To my surprise it worked! I need to collect all my notes and get right what changes are necessary before I write a detailed procedure for implementing SpamAssassin with MySQL. The Bayes mail tests seem to be working fine…no R/O tie failures as before.

Although I’m pleased DH made some upgrades, I was really unhappy those upgrades broke SpamAssassin. After SA broke, my SPAM went up to about 10 messages a day. Prior to that I was only getting 10 a week! I guess it shows the power of Bayesian filtering.

-Matt


#12

I set up MySQL the other day and it did not take too long, and has been working fine.