Matthew B-M: Mailfilter

Since I have the entire domain for colondot.net, I got fed up with getting junk mail, so I decided to write the following filter (you can see the whole thing here).

I employ several methods to do this:

• Automatic blacklist addresses
  If you send a message to one of these, the message bounces, your email address will be listed in a blacklist, and any future messages from that email address will bounce.
• Open Relay Blacklist
  I check against a number of the lists which contain lists of known open relays
• Host Blacklisting
  If your host has sent more than two emails that were marked as spam by other rules, then your host is marked as being bad, and all email from it will bounce.
• Domain Blacklist
  I have added a feature to manually allow the addition of sender domains to the sender blacklist
• One Use Addressing
  There exists a set of email addresses which are single use only. Once its been used, it is no longer a valid email address. I only ever have to recieve spam to it once.
• Time-limited email addresses
  This is a system where the email address only lasts until a particular date, which is coded into the email address itself. Then you don't have to remember to expire the address.
• Scoring Messages
  I have a list of pointers that something might be spam, but aren't definite. If the score gets too high, then it is rejected as spam.

This is all done through use of the Exim MTA, and it's capabilities of .forward files.

If you don't understand why this works, the documentation for exim filter files is at http://www.exim.org/exim-html-3.20/doc/html/filter.html

First of all, your .forward file must start with the following in order that exim recognises it should process it as a filter.

# Exim filter

In order to be useful, we decide that we're going to allow bounces back, these are usually useful, and then we can hope for reliable mail delivery.

# Allow mail delivery failures through
if error_message then
	finish
endif

RFCs 822 and 1123 mandate that we should accept postmaster for any domain:

# Allow postmaster access
if $original_local_part is postmaster then
	finish
endif

Automatic blacklist addresses

One method I employ is spambait addresses. If mail is sent to any of these, it is assumed to be spam, and the sender is immediately blacklisted. Any given host might also be blacklisted, but is given two chances.
(Please note: the regexps and spamtraps I use on this version of the page are different to the ones that I use for my own personal mail)

if ( ${lc:$original_local_part} is "spamdump" or
	${lc:$original_local_part} matches ^\\d?[a-z]{3}\\d+(-[a-z0-9+=-]*)?\$)
	then

determine that the mail is for a spambait

	seen mail expand file $home/mailfilter/spam-bounce
		to $return_path return message
		subject "Returned mail: blacklisted"

This will send back a mail saying that the person has been blacklisted. The expand keyword means that any $ variables can be expanded in the text, $sender_address is a possible one in this situation. This is a significant delivery because of seen.

	logfile $home/mailfilter/black.list 0644
	logwrite "${lc:$sender_address}: black"

add them to the blacklist.

	logfile $home/mailfilter/reject.log 0644
	logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: blacklisted"

Write a log of what we've done...

In here, there would also be some code for the host blacklisting, but I'll get to that below.

	finish
endif

stop any other bits of the filter file processing the delivery.

Having blacklisted the mail isn't much good if you can't do anything with the blacklist, so now we see why we wrote the blacklist out in that format. The following should appear near the top of the filter, to avoid entries being submitted twice.

if ("${lookup{${lc:$sender_address}} lsearch
	{$home/mailfilter/black.list}{$value}}" is "black")
	then

if this condition is true, they are in our blacklist

	seen mail expand file $home/mailfilter/bl-bounce
		to $return_path return message
		subject "Returned mail: blacklisted"

send them a bounce and tell them that they were already blacklisted

	logfile $home/mailfilter/reject.log 0644
	logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: already blacklisted"

log what we've done.

Again there is the same code for the host blacklisting as above

	finish
endif

end processing here.

Open Relay Blacklist

I also don't like any host that appears on a realtime blacklist, so:

if ( $header_X-RBL-Warning contains "mail-abuse.org" )
	then

These are put in by an rbl_domains=relays.mail-abuse.org/warn:relays.orbs.org/warn directive in the exim configuration file. This will return true if the host is on either of the lists.

	seen mail expand file $home/mailfilter/rbl-bounce
		to $return_path return message
		subject "Returned mail: blacklisted"

Send them a mail informing them that the service in question has realtime blacklisted them.

	logfile $home/mailfilter/reject.log 0644
	if ( $header_X-RBL-Warning contains "mail-abuse.org" ) then
		logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: on rbl (MAPS ONLY)"
	endif

And log the error.

This is enough to upgrade them from the host greylist to the blacklist, but not enough to put them on the greylist, see below.

	finish
endif

and stop any further deliveries

Host Blacklisting

As with the blacklisted mail addresses, there is a part that does the blacklisting, and a part which rejects if the mail is from a host on the blacklist.

The blacklisting happens in certain of the spam-trap rules - there are two lists, the black list and the grey list. If you are black then no mail is accepted from you. If you are grey, then one strike and you go onto the blacklist. This is done using the following code:

	if("${lookup{$sender_host_address} lsearch {$home/mailfilter/grey.servers}{$value}}" is "grey")
		then
		logwrite "[$tod_log] *** [$sender_host_address]: blacklisted host"
		logfile $home/mailfilter/black.servers 0644
		logwrite "$sender_host_address: black"
	endif

If you want to take something to grey or blacklist the host if it is already grey...

	if("${lookup{$sender_host_address} lsearch {$home/mailfilter/grey.servers}{$value}}" is "grey")
		then
		logwrite "[$tod_log] *** [$sender_host_address]: blacklisted host"
		logfile $home/mailfilter/black.servers 0644
		logwrite "$sender_host_address: black"
	else
		if("${lookup{$sender_host_address} lsearch {$home/mailfilter/grey.servers}{$value}}" is "")
		then
			logfile $home/mailfilter/grey.servers 0644
			logwrite "$sender_host_address: grey"
		endif
	endif

This allows you to mark hosts you trust by manually entering them as "<ip>: <something>" where <something> is not "grey" into the greylist. They will never get blacklisted if this is the case.

Once again you need the code to use the blacklist, which should go right at the top of the file to prevent you multi-blacklisting hosts.

if ("${lookup{$sender_host_address} lsearch {$home/mailfilter/black.servers}{$value}}" is "black")
	then
	seen mail expand file $home/mailfilter/blh-bounce
		to $return_path return message
		subject "Returned mail: blacklisted"
	logfile $home/mailfilter/reject.log 0644
	logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: host already blacklisted"
	finish
endif

Domain Blacklisting

This was something I added late on. It allows a matching in the domain part of the sender envelope. You add manually, through appending "<domain>: black" to the black.domains file.

if ("${lookup{${domain:${lc:$sender_address}}} lsearch {$home/mailfilter/black.domains}{$value}}" is "black")
	then
	seen mail expand file $home/mailfilter/bld-bounce
		to $return_path return message
		subject "Returned mail: blacklisted domain"
	logfile $home/mailfilter/reject.log 0644
	logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: already blacklisted domain"
	if("${lookup{$sender_host_address} lsearch {$home/mailfilter/grey.servers}{$value}}" is "grey")
		then
		logwrite "[$tod_log] *** [$sender_host_address]: blacklisted host"
		logfile $home/mailfilter/black.servers 0644
		logwrite "$sender_host_address: black"
	else
		if("${lookup{$sender_host_address} lsearch {$home/mailfilter/grey.servers}{$value}}" is "")
		then
			logfile $home/mailfilter/grey.servers 0644
			logwrite "$sender_host_address: grey"
		endif
	endif
	finish
endif

most of this is pretty similar to the blacklisting of email addresses, and as above, it includes the blacklist host code.

One Use Addresses

For non-spammers, but people I don't trust, I also want to have mail addresses which are marked as single-use, so that if I decide I want to get back in touch with people, I mail them using my proper address, but otherwise the emails just work the once. The method for doing this is very similar to the blacklisting, but this time we have a list called use.once with the $original_local_part variable in it.

if ( ${lc:$original_local_part} matches ^(^[a-z]{2}\\d{2}[a-z]{3}\\d{2})(-[a-z0-9+=-]*)?\$)
	then
	logfile $home/mailfilter/use.once 0644
	logwrite "${lc:$1}: used"
endif

As is evident, there is no significant delivery, so we've just effectively logged that we've seen that address used, and the email will still get delivered.

This is only the half of it, and you need:

if ( ${lc:$original_local_part} matches "^(.*?)(-[a-z0-9-+=]*)?\\\$" )
	then
	if ("${lookup{${lc:$1}} lsearch {$home/mailfilter/use.once}{$value}}" is "used")
		then
		seen mail expand file $home/mailfilter/used-bounce
			to $return_path return message
			subject "Returned mail: address no longer valid"
		logfile $home/mailfilter/reject.log 0644
		logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: reuse of a single-use address"
		finish
	endif
endif

To actually bounce the messages.

Time-limited Addresses

I also have some email addresses that have an expiry date encoded, obfuscated, within them.

The first problem is to find out today's date.

add 0 to n1
add 0 to n2
add 0 to n3

if ( $tod_log matches "^(\\\\d{4})-0*(\\\\d+)-0*(\\\\d)\\\\s" ) then
	add $1 to n1
	add $2 to n2
	add $3 to n3
endif

Now we'll extract the numbers as above.

if ( ${lc:$original_local_part} matches
	"^\\\\$0*(\\\\d+)\\\\$0*(\\\\d+)[a-z]0*(\\\\d+)\\\\d\\$(-[a-z0-9-+=]*)?\\\$" )
	then
	add $1 to n6
	add $2 to n4
	add $3 to n5

We will now add some random numbers to hide the fact that these are times.

	add 1977 to n4
	add -52 to n5
	add -9 to n6

And then check the mail for validity

	if ( ( $n1 is above $n4 ) or
		( $n1 is $n4 and $n2 is above $n5 ) or
		( $n1 is $n4 and $n2 is $n5 and $n3 is above $n6 ) or
		( $n4 is below 1 or $n5 is below 1 or $n6 is below 1 ) )
		then

If any of these tests succeeds, then the mail is not valid and should bounce.

		seen mail expand file $home/mailfilter/time-bounce
			to $return_path return message
			subject "Returned mail: address no longer valid"
		logfile $home/mailfilter/reject.log 0644
		logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: address timed out (expired $n4-$n5-$n6)"
		finish
	endif
endif

and finish, as before.

Scoring Messages

Finally, I have some pointers for messages that look like spam. Each time a rule succeeds, a counter is incremented, and if it ever reaches a certain value, the message fails and is bounced.

I use n9 as the variable for my scorefile, if it gets greater than 99, then the message is bounced. In some cases this takes two rules, in some cases 3.

if( ${lc:$message_body} matches "a\\\\shref=(['\"])mailto:([^@]@[^@])\$1" and
	$2 is not {$lc:$return_path} and $2 is not {$lc:$sender_address})
	then
	add 60 to n9
endif

If we find a "mailto:" link for which the address is not the same as the sender address or return path.

if (${lc:$message_body} matches "<html>")
	then
	add 60 to n9
endif

I don't often get real people sending me html mail.

if (${lc:$message_body} matches "<script")
	then
	add 90 to n9
endif

There has been an alarming tendency recently to send messages which are javascript encoded to get through these kind of scoring rules.

if (${lc:$message_body} matches "checks?\\\\spayable\\\\sto.*for \\\\\\$\\\\d+")
	then
	add 90 to n9
endif

No english person I know spells cheque as "check", so trap all the americans.

if($message_body matches "Dear Friend,")
	then
	add 80 to n9
endif

A very standard spam pointer

if(${lc:$header_subject} matches "\\\\s\\\\s\\\\s\\\\s+(\\\\(\\\\d+\\\\)|\\\\d+)\\\$")
	then
	add 40 to n9
endif

A lot of spams seem to have a subject which has a number (possibly in brackets) at the right-hand side, this is designed to catch this.

if(${lc:$header_to} matches "friend@" or ${lc:$header_from} matches "friend@")
	then
	add 90 to n9
endif

Classic spam pointer, but still occasionally seen.

if(${lc:$header_to} matches "@public.com")
	then
	add 90 to n9
endif

And this.

if(${lc:$message_body} matches "not spam" or ${lc:$message_body} matches "spam free")
	then
	add 80 to n9
endif

if(${lc:$message_body} matches "not junk mail")
	then
	add 80 to n9
endif

This has been appearing recently, especially in the disclaimers at the bottom. Genuine mail shouldn't say this kind of thing - it doesn't need to.

if(${lc:$message_body} matches "university diploma")
	then
	add 80 to n9
endif

Lots of these being advertised. Good thing I already have a degree. ;)

if(${domain:${lc:$header_to}} is "")
	then
	add 30 to n9
endif

Well, I certainly shouldn't see this in legitimate mails.

if(${lc:$sender_address} matches "([a-z\\\\d]+)@(lycos|hotmail|aol|yahoo|msn)\\\\.co(\\\\..*|m)" and
	$1 matches \\d)
	then
	add 40 to n9
endif

Don't trust anything with a number in the localpart at lycos, hotmail, aol, yahoo or msn.

if($header_subject matches \\\$\\\$+)
	then
	add 50 to n9
endif

Classic negative scorer in spam.

if(${lc:$sender_host_name} matches "ppp" or ${lc:$sender_host_name} matches "dial-?up")
	then
	add 50 to n9
endif

Catch things that should have been on the MAPS DUL, but apparently aren't.

if($n9 is above 99)
	then
	seen mail expand file $home/mailfilter/score-bounce
		to $return_path return message
		subject "Returned mail: failed score checking"
		logfile $home/mailfilter/reject.log 0644
		logwrite "[$tod_log] ${lc:$sender_address} [$sender_host_address] -> ${lc:$original_local_part}@${lc:$original_domain}: score failure"
	finish
endif

And check the score to see if it's too high. Bounce the message if it is.