1 This file documents some of the problems you may encounter when
2 upgrading dspam port. I will try my best to minimize these disruptions,
3 but sometimes they are unavoidable.
5 You should get into the habit of checking this file for changes each
6 time before attempting any port upgrade.
8 This file concetrate on the FreeBSD / port specific changes and one should read
9 also the UPGRADING enclosed in the dspam distribution. You can see it by doing
10 in the port directory:
11 make extract; more `find . -type f -maxdepth 2 -name UPGRADING`
13 ###########################################################################
14 # dspam-devel-3.10.1,1
17 Fix a problem with queries when using MySQL 5.5
20 ###########################################################################
21 # dspam-devel-3.10.0,1
27 ###########################################################################
28 # dspam-devel-3.10.0.r1,1
31 Various small fixes / improvements, see the CHANGELOG for details.
34 ###########################################################################
38 - Official "stable" release
39 - fixed message corruption problems with direct delivery (when using direct
40 delivery (e.g. DeliveryHost), certain servers require a linefeed after
41 carriage return otherwise the message will become malformatted).
42 - added a patch to fix CRLF pairs that are not correctly stripped out when
43 saving the message to the quarantine mbox file; useful if using LMTP or
44 SMTP (as opposed to running the client directly from a delivery script),
45 *AND* you are using the quarantine feature [1] I'm interested if anything
46 breaks in the WebUI if you're not using either SMTP or LMTP.
48 [1] patch from John Peacock <jpeacock@rowman.com>
49 http://mailing-list.nuclearelephant.com/6023.html
52 ###########################################################################
53 # dspam-devel-3.6.8.20061010.1118_1,1
57 - added safecat as LDA [1]
58 - give user the possibility to configure where dspam.conf is [1]
59 - bump PORTEPOCH to fix a stupid mistake (which had no real impact on the
60 port but there's no other way to do it - on 2006-12-17)
62 [1] patch from Emil Isberg <emil@pyttemjuk.se> [1]
65 ###########################################################################
66 # dspam-devel-3.6.8.20061010.1118
70 - fix longtime standing bug regarding CLAMAV: if "WITHOUT_CLAMAV_LOCAL=true"
71 clamav support wasn't compiled in
72 - added Cyrus's 2.3 deliver as LDA OPTION [1]
75 - removed oracle driver (unmaintained, didn't work on FreeBSD anyway)
78 - ability to add tagline to messages based on their classification; see
79 tagSpam and tagNonspam preferences in README
82 - fixed bugs from last release causing pgsql to fail on connection + other
84 - bugfix for segfault on mysql fail and null strlen calls for username
86 [1] ports/103910 for mail/dspam by Steffen Rick <srick@srick.org>
89 ###########################################################################
90 # dspam-devel-3.6.7.20060604.1555
95 - SupressWebStats is now WebStats in dspam.conf, and setting is inverted.
96 - The 'Tokenizer' setting in 3.8.0 replaces tokenizer definitions in the
97 "Feature" clause of previous version configurations. See dspam.conf.
98 - if using hash_drv storage driver, you should run cssconvert to upgrade
99 your .css files to a fully malaligned format.
100 - Earlier versions of 3.6 did not prepend a leading "-l" flag to specifying
101 log file selection. This is now required.
102 - renamed PValue "graham" to "bcr"
103 - dspam_train now works with filenames containing spaces
104 - +various optimizations to: tokenizer core, hash_drv driver (store offset
105 for writes), libdspam (preference lookups), osb/sbph
106 - HashMaxSeek defaults to 10
110 - ProcessorURLContext has been added to toggle whether URL specific tokens
111 are created in the tokenizer process. The "on" value is default for
112 previous versions of DSPAM.
113 - HashPctIncrease; see dspam.conf
114 - --client support for dspam_train
115 - cache runtime user information in daemon mode
116 - added logging of viruses (and the source)
119 - infinite loop on all non-preference-extension calls to delete a preference
120 - segfault when dlopen() to storage driver library fails
121 - segfault caused by bad use of va_args when vsyslog is called
122 - segfault in daemon mode when DeliveryHost or ClientHost is not specified
123 - added a connection check for pgsql, to reconnect on failure in daemon mode
124 - in hash_drv fixed some bugs which may have caused a crash
126 ###########################################################################
127 # dspam-devel-3.6.6 (stable release)
131 - Phased out deprecated Berkeley DB drivers
133 - when using logfile, write errors result in segfault
136 ###########################################################################
137 # dspam-devel-3.6.5.20060513.1125
141 - Fallback domains (see 2.6 in README)
143 - trainPristine preference turned "off" now overrides config turned "on"
144 - segfault fix for when correcting messages using UIDInSignature with MySQL
146 - dspam_corpus (replaced by newer dspam_train)
147 - dspam_genaliases (replaced by parse-to-headers, virtual users, etc)
150 ###########################################################################
151 # dspam-devel-3.6.5_1
154 Add dspam_debug rc.d options (default off): starts dspam with debug
155 logging (you need at least WITH_DEBUG).
156 Also fixed a few typos (like MIN_OPTIONS_VER value) in Makefile.
159 ###########################################################################
163 3.6.5 stable release, identical to our previous CVS snap detailed bellow.
166 ###########################################################################
167 # dspam-devel-3.6.4.20060421.2101
171 - invalid read/segfault (dspam.c)
172 - segfault on problems establishing connectivity to clamav (dspam.c)
173 - segfault on NULL username (mysql_drv.c)
174 - bug causing writing of flat-file preferences to fail (pref.c)
175 - fragment file overwritten on retrain (cgi)
176 - corrects the output of "dspam_admin aggr pref" (tools/dspam_admin.c)
177 - prevent quarantining of message when delivering summary
179 WebUI: a lot of fixes and improvements
181 PostgeSQL: improvements to purge scripts and object creation script
186 Add MySQL 5.1 as backend. [1]
187 Default MySQL version is now 5.0
189 Optionally depend on LightHTTPD instead of Apache if WITH_CGI=on [2]
191 Fix OPTIONS handling for INDEX (describe); this is a long standing bug
192 and a *big* pontyhat for me; it affected _only_ dependecies recorded in
193 INDEX _if_ OPTIONS were not set (like for package building); however,
194 the package itself was always built right, dependecies recorded OK, etc.
196 Add SCE-tindy as MASTER_SITE, much faster site that my other server. [3]
198 Requested by: Odhiambo WASHINGTON <odhiambo.raburu@wananchi.com> [1]
199 Submitted by: "Daniel S. Haischt"
200 <me@daniel.stefan.haischt.name> [2] (based on)
201 My thanks again to Joey Freeland for this machine and the bandwidth [3]
204 ###########################################################################
205 # dspam-devel-3.6.4.20060315.1518
209 - support for reading from multiple mysql servers; support for separate
210 read/write servers to be used with mysql_drv; see dspam.conf
211 - SYSLOG options knob: log to via syslog if "on" or in flat file if "off"
212 - dspam_stats: -t for displaying a total of all stats included in the
216 - fixed a bug causing --deliver=summary to return no output when used in
218 - fixed a segfault which can occur if TrainingMode is not specified in
222 ###########################################################################
223 # dspam-devel-3.6.4_1 (Added post-factum, mega libtool commit)
226 No functional changes, PORTREVISION bump for plist chage because now we
230 ###########################################################################
234 - dspam_train: support for training using an index file to define the order
237 - dspam_stats: OR --> OCA
240 ###########################################################################
241 # dspam-devel-3.6.3.20060211.0037
245 - mass-retraining option
248 - HashRec sizes prime numbers for better spread
251 ###########################################################################
252 # dspam-devel-3.6.3.20060208.1807
256 - undo option for retraining
257 - support for existing storeFragments option to recall message in history
260 - of all user preferences in the README.
263 ###########################################################################
264 # dspam-devel-3.6.3.20060203.1409
268 - accuracy fixes (esp. for FP dspam is much better that in 3.6.3 release)
269 - safeguards for segfault when strange clamav behavior
270 - for no output when using --classify in client/server mode
271 - LOG_WARNING if truncating MySQL user/pass + plus othersmall fixes
274 - dspam_train: a true training and testing mechanism, useful for building
275 pretrained databases or training a user with their own corpus. also provides
276 a test jig for measuring efficiency/accuracy with a corpus over a
278 - DeliveryHost for delivery to different hosts based on domain
279 - ClassAlias options, useful if classifying things other than spam
282 - removed neural networking (NEURAL_NET), which was experimental, needed a
283 rewrite, no support, and high maintenance
286 - changed names in dspam_stats; now displays TP (true positives),
287 TN (true negatives), FN (false negatives), and FP (false positives).
288 - dspam_corpus now uses default settings for features and training modes,
289 instead of its own and now requires --spam or --nonspam arguments
290 - the trainining buffer (Feature tb=n in dspam.conf) is now offby default
291 - no signature on classify
292 - ignore X-DSPAM-* in tokenizer code
295 ###########################################################################
299 BUGFIX: fixed bug causing "no trusted delivery agent configured" error when
300 calling dspam withot an agent configured, but not delivering - or when using
302 BUGFIX: fixed minor bug causing ChangeUserOnParse to format incorrectly
305 - rewrote some routines that were strdup'ing message body repeatedly
306 - changes to tokenization and probability assignment make a noticeable
307 difference in accuracy
308 - updated dspam_stats "-S" output to use more widely accepted readings:
309 SHR: Spam Hit Rate (true positive rate)
310 HSR: Ham Strike Rate (false positive rate)
311 OCA: Overall Classification Accuracy
312 - add large-scale support to dspam_logrotate
315 ###########################################################################
316 # dspam-devel-3.6.2_1 (2005-12-11 19:18)
319 Fix memory leak in BNR.
322 ###########################################################################
323 # dspam-devel-3.6.2 (2005-12-11 21:34)
326 Memory leak still here :(
328 BUGFIX: segfaults in daemon mode with hash_drv plus other hash_dvr fixes
329 BUGFIX: for how token values are calculated, cutting false positives nearly
331 BUGFIX: close connections to ldap after calls
332 on ldap failure fail database creation
334 CLAMAV* now only enables clam support in dspam; use CLAMAV_LOCAL to
335 RUN_DEPEND on the port and have it REQUIRED in the rc script.
336 Add MYSQL_LOCAL and POSTGRESQL_LOCAL options to RUN_DEPEND on servers and
337 have them REQUIRED in the rc script.
340 ###########################################################################
341 # dspam-devel-3.6.1 (2005-11-17)
344 fix RUN_DEPENDS for WITH_CGI=on by USE_APACHE=1.3+, no need to upgrade if
348 ###########################################################################
352 !!! WARNING: Both dspam 3.6.0 and 3.6.1 have a memory leak !!!
353 Watch the size and res (top -o size or top -o res) of dspam process and
354 restart it (/usr/local/etc/rc.d/dspam-devel.sh restart) when it gets to big.
356 BUGFIX: ParseToHeaders can segfault on malformatted To header
357 BUGFIX: TrackSources does not correctly parse messages with QMail Received
359 BUGFIX: Missing check for existing strcasestr
360 BUGFIX: X-DSPAM-Reclassified heading appears blank
361 BUGFIX: Plused-Detail support does not work with domains
362 BUGFIX: Hash databases are not 8-byte aligned (do not work with 64-bit systems)
363 use cssconvert to convert your hash file
364 BUGFIX: Certain daemon-mode operations append a trailing period
365 BUGFIX: DataSource 'document' option causes improper processing
366 BUGFIX: Segfaults occur on malformed Content-Type header
367 BUGFIX: History appears blank in dspam.cgi
369 - WITH_TRAIN_SCRIPT for an example training script dspam_train
372 ###########################################################################
376 - css_drv (CRM Sparse Spectra Storage Driver) is now called hash_drv,
377 hence: WITH_CSS -> WITH_HASH; also the max records number in hash_drv can
378 be set in dspam.conf; WARNING: the new file format is incompatble with the
379 old one but is considerable smaller
380 - you can now have the database drivers dynamically loaded at run time so
381 you can select multiple drivers in the OPTIONS screen; see dspam.conf.
382 - also algorithm selection is done exclusively via dspam.conf
383 - documented MySQL quoting bug in some versions of 4.1 (see mysql.txt),
384 and added MySQLSupressQuote option to compensate
385 - read UPGRADING and CHANGELOG and adjust your dspam.conf accordingly
386 - !!! WARNING !!! MaxMessageSize is also applied to ClamAV virus scanning.
387 Thanks to Patrick Hurrelmann <outi@bytephobia.de> for beta-testing and
388 catching a few mistakes.
391 ###########################################################################
392 # dspam-devel-3.5.2.20050725.0800
395 Please read carefully distributed docs like README, UPGRADING and CHANGES.
396 Notable new features:
398 Unified Spam Alias / UID in Signature Support
399 Integrated Clam A/V Support
400 Markovian Discrimination (CRM114)
401 CRM Sparse Spectra Storage Driver (css_drv)
403 - drop local patch to put X-DSPAM-User in all emails.
404 - some performance changes in PGSQL schema
405 - BUGFIX: SQLITE purge scrip for 3.x
408 ###########################################################################
409 # dspam-devel-3.5.0.20050512.1145
412 BUGFIX: fixed signature embedding on malformatted boundary bug
415 ###########################################################################
416 # dspam-devel-3.5.0.20050506.0930
419 - Support for clamd virus checking via TCP (see ClamAVHost, ClamAVPort and
420 ClamAVResponse in dspam.conf)
421 - Support "user+detail" inbound emails and "+quarantine" outbound emails
422 (see EnablePlusedDetail and QuarantineMailbox in dspam.conf)
424 - BUG FIX: "obscure" [1] bug causing dspam to crash under certain conditions
425 when the loose signature was provided without the appropriate delimiter
426 - BUG FIX: fix sqlite3 dependency check [2]
427 - BUG FIX: don't strip binary if we want GDB suport
428 - BUG FIX: DOMAIN_SCALE and LARGE_SCALE are incompatible catch this before
431 [1] "Obscure" = it took me about 4 hours and a 500 queue processed one by
432 one by hand to catch and reproduce the problem after 7 days of mail
434 [2] Pointed out by vanilla@ for mail/dspam but true here also
435 [3] Pointed out by Arvinn Løkkebakken <arvinn@whitebird.no> for mail/dspam
438 ###########################################################################
439 # dspam-devel-3.5.0.20050428.1100
442 This is the post-3.4 devel branch. A lot of things have changed and your
443 existing dspam.conf won't probably work anymore. Please compare it with the
444 dspam.conf.sample located in the same etc/ directory.
446 Most notable addition is the SMTP delivery which together with "standard"
447 inbound LMTP make a very nice content and light (e.g. it's the fastest setup
448 I've had so far) filter.
450 Database structure is roughly the same but you may wish to check your
451 existing database structure with the *.sql corresponding to you database
452 driver from EXAMPLESDIR (Mysql 4.1.x is now default as it's _much_ faster).
453 Please read carefully distributed docs like README, UPGRADING and CHANGES.
456 ###########################################################################
457 # dspam-devel-3.4.r2.20050305.0930 (this is post RC2A)
460 - BUGFIX: fix dspamc hanging when using --stdout
461 - BUGFIX: fixed TOE auto whitelisting
462 - BUGFIX: individual logging
463 - BUGFIX: parsetoheaders and daemon mode combo
464 - BUGFIX: autodetect bugs in cgi
466 - BUGFIXES for some un-standard mails segfaults
468 - NEW FEATURES: LMTP delivery, "standard" LMTP inbound support;
469 see CHANGELOG, README and dspam.conf for details (tested slightly on BSD)
471 - note (very late) SBL query support and SBL queue logging support
472 (both SBL client and server ports are WIP)
474 - As of 3.4 Beta 2, the PostgreSQL driver uses BIGINT instead of NUMERIC(20)
475 type to store tokens in database (fully backward compatible). This change
476 speedup driver and reduce storage space used by dspam. Use:
477 dspam_pg2int8 /tmp/dspam_pg_mig.sql
478 and follow the instructions.
481 ###########################################################################
482 # dspam-devel-3.4.r1.20050216.0945
485 - if dspam --daemon SIGBUS please send your sources date, CFLAGS and other
486 relevant information; this version is believed to be working; tested on:
487 i386 5.2.1-RELEASE-p1, i386 5.3-STABLE, amd64 5.3-RELEASE with help from Erwin Lansing <erwin@lansing.dk>,
488 "Daniel S. Haischt" <me@daniel.stefan.haischt.name> and Joey Freeland <joeyfreeland@gmail.com>
489 - patch dspam.c for bug causing signature to not get written to message
490 body (obtained from dspam cvs)
491 - patch dspam.c to write X-DSPAM-User also in innocent messages headers
492 - drop WITH_WHITELIST, it's not supported by ./configure but by dspam.conf
493 - (added post-factum) change POSTGRESQL* to POSTGRESQL because of "Split the
494 postgresql ports into a server and a client part"
497 ###########################################################################
501 The daemon mode seems to be completead now, so the port act like it should:
502 - if enabled with WITH_DEAMON we install the rc script; to start it at
503 boot put 'dspam_enable="YES"' in /etc/rc.conf and set the appropriate
504 options in dspam.conf
505 - read section "2.3 Client/Server Mode" from dspam README to learn how to
506 use dspam in daemon mode
507 - a simple portupgrade from previous versions will work with no problems
508 with your current setup, even if compiled with WITH_DEAMON (default)
510 - change default DB back-end to MySQL to make daemon mode work for package
512 - add MySQL v5.0.x support -- MYSQL50
513 (based on ports/74761 by Gea-Suan Lin <gslin@netnews.NCTU.edu.tw>)
514 - USE_MYSQL and have make describe output right
516 - change of name: SQLITE --> SQLITE2, as we now have sqlite3 support
517 - add Sqlite v3.x support -- SQLITE3
518 - add BNR_DEBUG option
519 - drop QUARANTINE_AGENT as is not supported by ./configure anymore and
520 set in dspam.conf UntrustedDeliveryAgent and QuarantineAgent to default to
521 TrustedDeliveryAgent, but leave them commented out
522 (ports/75548 by Julien Gabel)
523 - change default DSPAM_HOME_MODE to 0770
524 (ports/75549 by Julien Gabel <jpeg@thilelli.net>)
525 - DSPAM_HOME_{OWNER,GROUP} default to DSPAM_{OWNER,GROUP}
526 - default maximum message size to 300KB in dspam.conf
527 - correct long-standing copy/paste bug (DSPAM_{GROUP,MODE} were set to
528 DSPAM_HOME_{GROUP,MODE})
529 - USE_MYSQL and have make describe output right
530 - add support for exim [2] which we build with the same database back-end
531 as dspam if possible ( suggested, incorrectly, in ports/74762 by
532 Shahbaz Javeed <sjaveed@adderpit.com> )
533 - fix PREF_EXT and BDB4 cases as well in our Makefile
536 #############################################################################
537 # dspam-devel=3.3.0.a.20041203.1245_1 (entry added post-factum, it was an
538 # automated update approved by me but not under my control)
541 - chase dependecy for sqlite
544 ############################################################################
545 # dspam-devel-3.3.0.a.20041201.1330
548 Say hello to the daemonized dspam! Enble it at compile-time with DAEMON option.
552 MAIL FROM: [password]@[server id]
553 RCPT TO: [list of DSPAM users] [DSPAM parameters] [delivery agent parameters]
558 - daemon processes *and* delivers at the moment (including to stdout).
559 - For client-side delivery, use --stdout and configure your MTA to deliver.
560 - it may be necessary to perform all delivery in the client, as using fork &&
561 pipe in a MT app may be bad
562 - implemented multiple responses per recipient
563 - changed storage driver layer to use a driver context (for caching connections)
564 - stateful db connections done for mysql driver
567 - remember that all this is experimental and it may change at any time.
570 ############################################################################
571 # dspam-devel-3.3.0.a.20041125.0945
574 - changed the naming scheme because of to many upstream modification in
575 tarball naming - right now it lacks any version (for the same reason the
576 distfile is renamed from snapshot.tar.gz to PKGNAME and sits on my server);
577 \ it is now made up of:
578 PORTNAME: dspam-devel
579 next MINOR_VERSION: 3.3.0
581 SNAPSHOT_DATE: 20041125.0945
582 \ the strange ``a'' is necessary for pkg_version to work correctly when we will
583 have the 3.3.0 beta's, rc's and release (tested with pkg_version -t)
585 - follow the update procedure from the entry bellow
587 - there is a new and very useful dspam.conf option:
588 MaxMessageSize - specify a maximum message size to process
589 \ I suggest setting it somewhere between 100 and 300 KB, it is unlikely the
590 spam you are receiving is bigger and this will improve performance as big
591 messages won't stuck the database
593 - tum mode has been enhanced and it should outperform teft
595 ###############################################################################
596 # dspam-devel-3.2.0-20041027.0930 (never comited in the cvs, don't ask me why):
599 - dspam configuration is now done via dspam.conf
601 - please remove any OPTION
603 - the following OPTIONS should be set now in dspam.conf
604 OPTION .conf name def. value
605 -------------------------------------------------------------------------------
606 DSPAM_HOME Home /var/dspam
607 WITH_*_LDA TrustedDeliveryAgent mail.local
608 UntrustedDeliveryAgent /usr/libexec/mail.local d %u
609 QUARANTINE_AGENT QuarantineAgent #
611 WITH_SPAM_SUBJ Preference "spamAction=tag" #
612 Preference "spamSubject=SPAM" #
613 USER_LOGGING UserLog on
614 SYSTEM_LOGGING SystemLog on
615 WEBMAIL TrainPristine off
617 SAT TrackSources spam ham #
618 PARSE_TO_HEADERS ParseToHeaders on
619 BROKEN_MTA Broken lineStripping #
620 BROKEN_ERR_CODES Broken returnCodes #
621 SIGNATURE_HEADERS Preference "signatureLocation= message" # headers
623 - the following OPTIONS should does not exist anymore:
624 SIGNATURE_ATACH WITH_HOMEDIR_DOT
626 - the following OPTIONS have their name changed to follow vedor names
627 TRAD_BAYES GRAHAM_BAYES
628 ALT_BAYES BURTON_BAYES
629 WITH_SIGNATURE_LIFE SIGNATURE_LIFE
633 ----------------------------------------------
634 LOG_DIR /var/log/dspam
643 To upgrade you should:
645 0. Read docs enclosed with teh dpsam distribution, beginnig with UPGRADING and
648 1. Adjust your OPTIONs
649 ( make rmconfig config in dspam port dir, command line, pkg_tools.conf )
652 in dspam port dir with appropiate OPTIONS adn VARs (see changes above)
654 3. stop you MTA or deactivate dspam from it
656 5. make VARs........ install
658 4.a) If using MySQL 4.0.xx or PostgreSQL:
659 - back-up the database
660 - use the appropiate migration script from files dir of dspam port dir
661 (if using MySQL 4.1.x or 3.xx you're on your own for now)
663 5. Back-up and them move the files from the old DSPAM_HOME
664 (defaulted: /usr/loca/etc/dspam) to the new DSPAM_HOME (default /vad/db/dspam)
665 or where ever you defined DSPA_HOME in 2 above.
667 6. Adjust you new dspam.conf according with your setup OPTIONS and VARs defined
668 on make / mak install step.
670 7. Remove the old setup files (you have them in the back-up, right ?)
672 7. Enable dspam in the MTA setup, but only locally or something and start testing.
674 If something ain't working is one of:
675 - permissions problems (hint: with what UID
676 is dspam called from your MTA vs. DSPAM_HOME and DSPAM_LOG_DIR)
677 - something wrong in dspam.conf (hint: you didn't remove old config files
678 and they are still used ?)