This file documents some of the problems you may encounter when upgrading dspam port. I will try my best to minimize these disruptions, but sometimes they are unavoidable. You should get into the habit of checking this file for changes each time before attempting any port upgrade. This file concetrate on the FreeBSD / port specific changes and one should read also the UPGRADING enclosed in the dspam distribution. You can see it by doing in the port directory: make extract; more `find . -type f -maxdepth 2 -name UPGRADING` ########################################################################### # dspam-devel-3.10.1,1 # Fix a problem with queries when using MySQL 5.5 ########################################################################### # dspam-devel-3.10.0,1 # No change from RC1. ########################################################################### # dspam-devel-3.10.0.r1,1 # Various small fixes / improvements, see the CHANGELOG for details. ########################################################################### # dspam-devel-3.8.0,1 # - Official "stable" release - fixed message corruption problems with direct delivery (when using direct delivery (e.g. DeliveryHost), certain servers require a linefeed after carriage return otherwise the message will become malformatted). - added a patch to fix CRLF pairs that are not correctly stripped out when saving the message to the quarantine mbox file; useful if using LMTP or SMTP (as opposed to running the client directly from a delivery script), *AND* you are using the quarantine feature [1] I'm interested if anything breaks in the WebUI if you're not using either SMTP or LMTP. [1] patch from John Peacock http://mailing-list.nuclearelephant.com/6023.html ########################################################################### # dspam-devel-3.6.8.20061010.1118_1,1 # Port changes: - added safecat as LDA [1] - give user the possibility to configure where dspam.conf is [1] - bump PORTEPOCH to fix a stupid mistake (which had no real impact on the port but there's no other way to do it - on 2006-12-17) [1] patch from Emil Isberg [1] ########################################################################### # dspam-devel-3.6.8.20061010.1118 # Port changes: - fix longtime standing bug regarding CLAMAV: if "WITHOUT_CLAMAV_LOCAL=true" clamav support wasn't compiled in - added Cyrus's 2.3 deliver as LDA OPTION [1] Change features: - removed oracle driver (unmaintained, didn't work on FreeBSD anyway) New features: - ability to add tagline to messages based on their classification; see tagSpam and tagNonspam preferences in README Bug fixes: - fixed bugs from last release causing pgsql to fail on connection + other fixes - bugfix for segfault on mysql fail and null strlen calls for username [1] ports/103910 for mail/dspam by Steffen Rick ########################################################################### # dspam-devel-3.6.7.20060604.1555 # Change features: - API change !! - SupressWebStats is now WebStats in dspam.conf, and setting is inverted. - The 'Tokenizer' setting in 3.8.0 replaces tokenizer definitions in the "Feature" clause of previous version configurations. See dspam.conf. - if using hash_drv storage driver, you should run cssconvert to upgrade your .css files to a fully malaligned format. - Earlier versions of 3.6 did not prepend a leading "-l" flag to specifying log file selection. This is now required. - renamed PValue "graham" to "bcr" - dspam_train now works with filenames containing spaces - +various optimizations to: tokenizer core, hash_drv driver (store offset for writes), libdspam (preference lookups), osb/sbph - HashMaxSeek defaults to 10 New features: - OSB tokenizer - ProcessorURLContext has been added to toggle whether URL specific tokens are created in the tokenizer process. The "on" value is default for previous versions of DSPAM. - HashPctIncrease; see dspam.conf - --client support for dspam_train - cache runtime user information in daemon mode - added logging of viruses (and the source) Bug fixes: - infinite loop on all non-preference-extension calls to delete a preference - segfault when dlopen() to storage driver library fails - segfault caused by bad use of va_args when vsyslog is called - segfault in daemon mode when DeliveryHost or ClientHost is not specified - added a connection check for pgsql, to reconnect on failure in daemon mode - in hash_drv fixed some bugs which may have caused a crash ########################################################################### # dspam-devel-3.6.6 (stable release) # Removed features: - Phased out deprecated Berkeley DB drivers Bugfixes: - when using logfile, write errors result in segfault ########################################################################### # dspam-devel-3.6.5.20060513.1125 # New feature: - Fallback domains (see 2.6 in README) Bug fixes: - trainPristine preference turned "off" now overrides config turned "on" - segfault fix for when correcting messages using UIDInSignature with MySQL Removed features: - dspam_corpus (replaced by newer dspam_train) - dspam_genaliases (replaced by parse-to-headers, virtual users, etc) ########################################################################### # dspam-devel-3.6.5_1 # Add dspam_debug rc.d options (default off): starts dspam with debug logging (you need at least WITH_DEBUG). Also fixed a few typos (like MIN_OPTIONS_VER value) in Makefile. ########################################################################### # dspam-devel-3.6.5 # 3.6.5 stable release, identical to our previous CVS snap detailed bellow. ########################################################################### # dspam-devel-3.6.4.20060421.2101 # Bug-fixes: - invalid read/segfault (dspam.c) - segfault on problems establishing connectivity to clamav (dspam.c) - segfault on NULL username (mysql_drv.c) - bug causing writing of flat-file preferences to fail (pref.c) - fragment file overwritten on retrain (cgi) - corrects the output of "dspam_admin aggr pref" (tools/dspam_admin.c) - prevent quarantining of message when delivering summary WebUI: a lot of fixes and improvements PostgeSQL: improvements to purge scripts and object creation script Port changes: ------------- Add MySQL 5.1 as backend. [1] Default MySQL version is now 5.0 Optionally depend on LightHTTPD instead of Apache if WITH_CGI=on [2] Fix OPTIONS handling for INDEX (describe); this is a long standing bug and a *big* pontyhat for me; it affected _only_ dependecies recorded in INDEX _if_ OPTIONS were not set (like for package building); however, the package itself was always built right, dependecies recorded OK, etc. Add SCE-tindy as MASTER_SITE, much faster site that my other server. [3] Requested by: Odhiambo WASHINGTON [1] Submitted by: "Daniel S. Haischt" [2] (based on) My thanks again to Joey Freeland for this machine and the bandwidth [3] ########################################################################### # dspam-devel-3.6.4.20060315.1518 # New feature: - support for reading from multiple mysql servers; support for separate read/write servers to be used with mysql_drv; see dspam.conf - SYSLOG options knob: log to via syslog if "on" or in flat file if "off" - dspam_stats: -t for displaying a total of all stats included in the original query Bugfixes: - fixed a bug causing --deliver=summary to return no output when used in dspamc - fixed a segfault which can occur if TrainingMode is not specified in dspam.conf ########################################################################### # dspam-devel-3.6.4_1 (Added post-factum, mega libtool commit) # No functional changes, PORTREVISION bump for plist chage because now we install .la files. ########################################################################### # dspam-devel-3.6.4 # - dspam_train: support for training using an index file to define the order of ham/spam - dspam_stats: OR --> OCA ########################################################################### # dspam-devel-3.6.3.20060211.0037 # WebUI: - mass-retraining option dspam.conf: - HashRec sizes prime numbers for better spread ########################################################################### # dspam-devel-3.6.3.20060208.1807 # WebUI: - undo option for retraining - support for existing storeFragments option to recall message in history Documentation: - of all user preferences in the README. ########################################################################### # dspam-devel-3.6.3.20060203.1409 # BugFixes: - accuracy fixes (esp. for FP dspam is much better that in 3.6.3 release) - safeguards for segfault when strange clamav behavior - for no output when using --classify in client/server mode - LOG_WARNING if truncating MySQL user/pass + plus othersmall fixes New features: - dspam_train: a true training and testing mechanism, useful for building pretrained databases or training a user with their own corpus. also provides a test jig for measuring efficiency/accuracy with a corpus over a configuration. - DeliveryHost for delivery to different hosts based on domain - ClassAlias options, useful if classifying things other than spam Removed features: - removed neural networking (NEURAL_NET), which was experimental, needed a rewrite, no support, and high maintenance Changed features: - changed names in dspam_stats; now displays TP (true positives), TN (true negatives), FN (false negatives), and FP (false positives). - dspam_corpus now uses default settings for features and training modes, instead of its own and now requires --spam or --nonspam arguments - the trainining buffer (Feature tb=n in dspam.conf) is now offby default - no signature on classify - ignore X-DSPAM-* in tokenizer code ########################################################################### # dspam-devel-3.6.3 # BUGFIX: fixed bug causing "no trusted delivery agent configured" error when calling dspam withot an agent configured, but not delivering - or when using --classify BUGFIX: fixed minor bug causing ChangeUserOnParse to format incorrectly Notable changes: - rewrote some routines that were strdup'ing message body repeatedly - changes to tokenization and probability assignment make a noticeable difference in accuracy - updated dspam_stats "-S" output to use more widely accepted readings: SHR: Spam Hit Rate (true positive rate) HSR: Ham Strike Rate (false positive rate) OCA: Overall Classification Accuracy - add large-scale support to dspam_logrotate ########################################################################### # dspam-devel-3.6.2_1 (2005-12-11 19:18) # Fix memory leak in BNR. ########################################################################### # dspam-devel-3.6.2 (2005-12-11 21:34) # Memory leak still here :( BUGFIX: segfaults in daemon mode with hash_drv plus other hash_dvr fixes BUGFIX: for how token values are calculated, cutting false positives nearly in half BUGFIX: close connections to ldap after calls on ldap failure fail database creation CLAMAV* now only enables clam support in dspam; use CLAMAV_LOCAL to RUN_DEPEND on the port and have it REQUIRED in the rc script. Add MYSQL_LOCAL and POSTGRESQL_LOCAL options to RUN_DEPEND on servers and have them REQUIRED in the rc script. ########################################################################### # dspam-devel-3.6.1 (2005-11-17) # fix RUN_DEPENDS for WITH_CGI=on by USE_APACHE=1.3+, no need to upgrade if already installed ########################################################################### # dspam-devel-3.6.1 # !!! WARNING: Both dspam 3.6.0 and 3.6.1 have a memory leak !!! Watch the size and res (top -o size or top -o res) of dspam process and restart it (/usr/local/etc/rc.d/dspam-devel.sh restart) when it gets to big. BUGFIX: ParseToHeaders can segfault on malformatted To header BUGFIX: TrackSources does not correctly parse messages with QMail Received headers BUGFIX: Missing check for existing strcasestr BUGFIX: X-DSPAM-Reclassified heading appears blank BUGFIX: Plused-Detail support does not work with domains BUGFIX: Hash databases are not 8-byte aligned (do not work with 64-bit systems) use cssconvert to convert your hash file BUGFIX: Certain daemon-mode operations append a trailing period BUGFIX: DataSource 'document' option causes improper processing BUGFIX: Segfaults occur on malformed Content-Type header BUGFIX: History appears blank in dspam.cgi - WITH_TRAIN_SCRIPT for an example training script dspam_train ########################################################################### # dspam-devel-3.6.0 # - css_drv (CRM Sparse Spectra Storage Driver) is now called hash_drv, hence: WITH_CSS -> WITH_HASH; also the max records number in hash_drv can be set in dspam.conf; WARNING: the new file format is incompatble with the old one but is considerable smaller - you can now have the database drivers dynamically loaded at run time so you can select multiple drivers in the OPTIONS screen; see dspam.conf. - also algorithm selection is done exclusively via dspam.conf - documented MySQL quoting bug in some versions of 4.1 (see mysql.txt), and added MySQLSupressQuote option to compensate - read UPGRADING and CHANGELOG and adjust your dspam.conf accordingly - !!! WARNING !!! MaxMessageSize is also applied to ClamAV virus scanning. Thanks to Patrick Hurrelmann for beta-testing and catching a few mistakes. ########################################################################### # dspam-devel-3.5.2.20050725.0800 # Please read carefully distributed docs like README, UPGRADING and CHANGES. Notable new features: LDAP Verification Unified Spam Alias / UID in Signature Support Integrated Clam A/V Support Markovian Discrimination (CRM114) CRM Sparse Spectra Storage Driver (css_drv) - drop local patch to put X-DSPAM-User in all emails. - some performance changes in PGSQL schema - BUGFIX: SQLITE purge scrip for 3.x ########################################################################### # dspam-devel-3.5.0.20050512.1145 # BUGFIX: fixed signature embedding on malformatted boundary bug ########################################################################### # dspam-devel-3.5.0.20050506.0930 # - Support for clamd virus checking via TCP (see ClamAVHost, ClamAVPort and ClamAVResponse in dspam.conf) - Support "user+detail" inbound emails and "+quarantine" outbound emails (see EnablePlusedDetail and QuarantineMailbox in dspam.conf) - BUG FIX: "obscure" [1] bug causing dspam to crash under certain conditions when the loose signature was provided without the appropriate delimiter - BUG FIX: fix sqlite3 dependency check [2] - BUG FIX: don't strip binary if we want GDB suport - BUG FIX: DOMAIN_SCALE and LARGE_SCALE are incompatible catch this before ./cofigure fails [3] [1] "Obscure" = it took me about 4 hours and a 500 queue processed one by one by hand to catch and reproduce the problem after 7 days of mail processing w/o error [2] Pointed out by vanilla@ for mail/dspam but true here also [3] Pointed out by Arvinn Løkkebakken for mail/dspam but true here also ########################################################################### # dspam-devel-3.5.0.20050428.1100 # This is the post-3.4 devel branch. A lot of things have changed and your existing dspam.conf won't probably work anymore. Please compare it with the dspam.conf.sample located in the same etc/ directory. Most notable addition is the SMTP delivery which together with "standard" inbound LMTP make a very nice content and light (e.g. it's the fastest setup I've had so far) filter. Database structure is roughly the same but you may wish to check your existing database structure with the *.sql corresponding to you database driver from EXAMPLESDIR (Mysql 4.1.x is now default as it's _much_ faster). Please read carefully distributed docs like README, UPGRADING and CHANGES. ########################################################################### # dspam-devel-3.4.r2.20050305.0930 (this is post RC2A) # - BUGFIX: fix dspamc hanging when using --stdout - BUGFIX: fixed TOE auto whitelisting - BUGFIX: individual logging - BUGFIX: parsetoheaders and daemon mode combo - BUGFIX: autodetect bugs in cgi - BUGFIXES for stats - BUGFIXES for some un-standard mails segfaults - NEW FEATURES: LMTP delivery, "standard" LMTP inbound support; see CHANGELOG, README and dspam.conf for details (tested slightly on BSD) - note (very late) SBL query support and SBL queue logging support (both SBL client and server ports are WIP) - As of 3.4 Beta 2, the PostgreSQL driver uses BIGINT instead of NUMERIC(20) type to store tokens in database (fully backward compatible). This change speedup driver and reduce storage space used by dspam. Use: dspam_pg2int8 /tmp/dspam_pg_mig.sql and follow the instructions. ########################################################################### # dspam-devel-3.4.r1.20050216.0945 # - if dspam --daemon SIGBUS please send your sources date, CFLAGS and other relevant information; this version is believed to be working; tested on: i386 5.2.1-RELEASE-p1, i386 5.3-STABLE, amd64 5.3-RELEASE with help from Erwin Lansing , "Daniel S. Haischt" and Joey Freeland - patch dspam.c for bug causing signature to not get written to message body (obtained from dspam cvs) - patch dspam.c to write X-DSPAM-User also in innocent messages headers - drop WITH_WHITELIST, it's not supported by ./configure but by dspam.conf - (added post-factum) change POSTGRESQL* to POSTGRESQL because of "Split the postgresql ports into a server and a client part" ########################################################################### # dspam-devel-3.3.4 # The daemon mode seems to be completead now, so the port act like it should: - if enabled with WITH_DEAMON we install the rc script; to start it at boot put 'dspam_enable="YES"' in /etc/rc.conf and set the appropriate options in dspam.conf - read section "2.3 Client/Server Mode" from dspam README to learn how to use dspam in daemon mode - a simple portupgrade from previous versions will work with no problems with your current setup, even if compiled with WITH_DEAMON (default) - change default DB back-end to MySQL to make daemon mode work for package building - add MySQL v5.0.x support -- MYSQL50 (based on ports/74761 by Gea-Suan Lin ) - USE_MYSQL and have make describe output right - change of name: SQLITE --> SQLITE2, as we now have sqlite3 support - add Sqlite v3.x support -- SQLITE3 - add BNR_DEBUG option - drop QUARANTINE_AGENT as is not supported by ./configure anymore and set in dspam.conf UntrustedDeliveryAgent and QuarantineAgent to default to TrustedDeliveryAgent, but leave them commented out (ports/75548 by Julien Gabel) - change default DSPAM_HOME_MODE to 0770 (ports/75549 by Julien Gabel ) - DSPAM_HOME_{OWNER,GROUP} default to DSPAM_{OWNER,GROUP} - default maximum message size to 300KB in dspam.conf - correct long-standing copy/paste bug (DSPAM_{GROUP,MODE} were set to DSPAM_HOME_{GROUP,MODE}) - USE_MYSQL and have make describe output right - add support for exim [2] which we build with the same database back-end as dspam if possible ( suggested, incorrectly, in ports/74762 by Shahbaz Javeed ) - fix PREF_EXT and BDB4 cases as well in our Makefile ############################################################################# # dspam-devel=3.3.0.a.20041203.1245_1 (entry added post-factum, it was an # automated update approved by me but not under my control) # - chase dependecy for sqlite ############################################################################ # dspam-devel-3.3.0.a.20041201.1330 # Say hello to the daemonized dspam! Enble it at compile-time with DAEMON option. - it speaks LMTP: LHLO MAIL FROM: [password]@[server id] RCPT TO: [list of DSPAM users] [DSPAM parameters] [delivery agent parameters] DATA [message] . - daemon processes *and* delivers at the moment (including to stdout). - For client-side delivery, use --stdout and configure your MTA to deliver. - it may be necessary to perform all delivery in the client, as using fork && pipe in a MT app may be bad - implemented multiple responses per recipient - changed storage driver layer to use a driver context (for caching connections) - stateful db connections done for mysql driver - to do next: client - remember that all this is experimental and it may change at any time. ############################################################################ # dspam-devel-3.3.0.a.20041125.0945 # - changed the naming scheme because of to many upstream modification in tarball naming - right now it lacks any version (for the same reason the distfile is renamed from snapshot.tar.gz to PKGNAME and sits on my server); \ it is now made up of: PORTNAME: dspam-devel next MINOR_VERSION: 3.3.0 we're in pre-beta: a SNAPSHOT_DATE: 20041125.0945 \ the strange ``a'' is necessary for pkg_version to work correctly when we will have the 3.3.0 beta's, rc's and release (tested with pkg_version -t) - follow the update procedure from the entry bellow - there is a new and very useful dspam.conf option: MaxMessageSize - specify a maximum message size to process \ I suggest setting it somewhere between 100 and 300 KB, it is unlikely the spam you are receiving is bigger and this will improve performance as big messages won't stuck the database - tum mode has been enhanced and it should outperform teft ############################################################################### # dspam-devel-3.2.0-20041027.0930 (never comited in the cvs, don't ask me why): # - dspam configuration is now done via dspam.conf - please remove any OPTION - the following OPTIONS should be set now in dspam.conf OPTION .conf name def. value ------------------------------------------------------------------------------- DSPAM_HOME Home /var/dspam WITH_*_LDA TrustedDeliveryAgent mail.local UntrustedDeliveryAgent /usr/libexec/mail.local d %u QUARANTINE_AGENT QuarantineAgent # WITH_SPAM_SUBJ Preference "spamAction=tag" # Preference "spamSubject=SPAM" # USER_LOGGING UserLog on SYSTEM_LOGGING SystemLog on WEBMAIL TrainPristine off WITH_OPT_IN Opt out SAT TrackSources spam ham # PARSE_TO_HEADERS ParseToHeaders on BROKEN_MTA Broken lineStripping # BROKEN_ERR_CODES Broken returnCodes # SIGNATURE_HEADERS Preference "signatureLocation= message" # headers - the following OPTIONS should does not exist anymore: SIGNATURE_ATACH WITH_HOMEDIR_DOT - the following OPTIONS have their name changed to follow vedor names TRAD_BAYES GRAHAM_BAYES ALT_BAYES BURTON_BAYES WITH_SIGNATURE_LIFE SIGNATURE_LIFE - new OPTIONS OPTION port default ---------------------------------------------- LOG_DIR /var/log/dspam WITH_USER_HOMEDIR DSPAM_HOME_OWNER DSPAM_HOME_GROUP DSPAM_HOME_MODE DSPAM_MODE DSPAM_OWNER DSPAM_GROUP To upgrade you should: 0. Read docs enclosed with teh dpsam distribution, beginnig with UPGRADING and README 1. Adjust your OPTIONs ( make rmconfig config in dspam port dir, command line, pkg_tools.conf ) 2. make VARs....... in dspam port dir with appropiate OPTIONS adn VARs (see changes above) 3. stop you MTA or deactivate dspam from it 5. make VARs........ install 4.a) If using MySQL 4.0.xx or PostgreSQL: - back-up the database - use the appropiate migration script from files dir of dspam port dir (if using MySQL 4.1.x or 3.xx you're on your own for now) 5. Back-up and them move the files from the old DSPAM_HOME (defaulted: /usr/loca/etc/dspam) to the new DSPAM_HOME (default /vad/db/dspam) or where ever you defined DSPA_HOME in 2 above. 6. Adjust you new dspam.conf according with your setup OPTIONS and VARs defined on make / mak install step. 7. Remove the old setup files (you have them in the back-up, right ?) 7. Enable dspam in the MTA setup, but only locally or something and start testing. If something ain't working is one of: - permissions problems (hint: with what UID is dspam called from your MTA vs. DSPAM_HOME and DSPAM_LOG_DIR) - something wrong in dspam.conf (hint: you didn't remove old config files and they are still used ?)