Jan 212007

Here’s some late-night brainstorming on what can be done with spamd greylist entries during the time they’re greylisted.

spamd(8) is OpenBSD’s spam deferral daemon, which manipulates pf(4) tables to pass or block network connections from email senders to your valid mail servers. spamd was written by Bob Beck, and it’s worth listening to his NYCBSDCON presentation on spamd, which is a very funny presentation.

Greylisting is the practice of monitoring the envelope from and to, and the sending IP address — a ‘tuple’ — and temporarily rejecting each new, unique instance. Spam-sending software has historically not done what real mail servers do on a temporary failure — retry later — so greylisting has been a computationally cheap means of ignoring a lot of spam. It’s like a Rabbi’s traditional three-time rebuff of a person seeking to convert to Judaism. If they’re serious, they’ll be back. Every time a new IP address attempts to establish an SMTP session with your mailserver, reject the message temporarily, and if they come back, carry on with mail processing.

Greylisting as originally described only whitelisted a distinct tuple. For instance, if john@example.com sent email to suzy@example.edu from IP a.b.c.d, john’s mail server would get temporarily rejected, would try again later, and after a set interval, his email would then be allowed to be delivered. Now he was whitelisted. He could send suzy@ another message, and it would be delivered immediately, provided it came from the same IP address. He’d have to go through the whole process again, though, to send mail to the different address becky@example.com.

spamd does no mail processing. It doesn’t forward mail. It doesn’t receive any mail. spamd will never once handle a single SMTP session that will result in a successful delivery. It’s a minimalist SMTP implementation that elicits the needed tuple, and then gives a temporary error. If you successfully retry after the threshold time, you’re whitelisted. You can send any from or to from that IP address now. Since spamd only manipulates IP addresses in a firewall table, it can’t analyze the body of the message and won’t even see further SMTP sessions from that address.

spamd manipulates pf firewall tables. When an SMTP session is attempted to your mailserver: if the IP is whitelisted, then it connects to your server; if the IP is not whitelisted, the connection gets re-routed to the spamd daemon, which wastes a little of the sender’s time (a good thing) and adds entries to the greylisting database.

The best part of greylisting is that it allows you some check-up time on email from new hosts. Take their name and who they’re here to see, and tell them to come back later. In the meantime, do some calling around to see if they’re legit.

After implementing spamd in greylisting mode, I started making notes about different ways to filter out even more spam, based only on the nine fields in a GREY entry, as Beck’s greyscanner does. There is, by default, 26 minutes from the time we first hear from a new IP address until we’ll let them connect to our real mailserver. During that 26 minutes, we can do some sleuthing to see if we should just cut them out altogether, or whitelist them. The longer our greylist passtime, the more time different services (DNSBL’s, SURBL‘s, etc.,) have to find out about the IP address in question.

The spamd database entries for greylisted hosts (this is the first time we’ve heard from you) look like this, as processed with the spamdb(8) utility:

type|source ip|from|to|first|pass|expire|block|pass

For example:

[Note: angle brackets removed from from and to, because this editor that comes with WordPress just sucks. Always munges angle brackets. Manually enter the HTML for gt and lt, and it still munges them. Very irritating.]

All greylisted entries are of type “GREY,” so `spamdb | grep “^GREY”` will list all greylisted entries. What to do with them…? Can we make reasonable determinations about the validity of these email attempts using only these 9 fields?

type is already known. We could deduce important information from source ip, from, to, and first. pass, expire, block, and pass are administrator-configuration variable or random, so I see nothing that could be gleaned from them.

With these four pieces of information, we have fifteen possible combinations of criteria (2n-1). We can whitelist or trap with spamdb. It would be nice to be able to manipulate passtime to extend the greylist threshold. Say, default is 26 minutes, and the max passtime is 210 minutes (for instance) — I’d like to be able to `spamdb -m a.b.c.d` to maximize the time before the IP can get whitelisted. I’d provide the diff if I knew C better….

source ip: [“We have a message from IP address __.__.__.__. Based solely on that, what should we do?”]

  • Whitelisted hosts won’t get routed to spamd, so you’d never whitelist on this — it should already be done.
  • See if they’re on a DNSBL. I highly recommend the standard sbl-xbl.spamhaus.org list. If the sender is on the spamhaus list, it is an overwhelming indication that I don’t want them to access my real server. I do not accept any mail from anyone on the spamhaus list.
  • Check against a similar reputation service. Example: Borderware Security Network.
  • Geolocation. China and Korea IP addresses are routinely blacklisted, since, historically, they’ve been havens for spammers. And it’s very doubtful a regional plumbing distributor in Akron would be getting mail from a Nigerian mail server. Perl module IP::Country::DNSBL – IP geolocation via DNS. Personally, I would be wary of this, given the global nature of bigger business, but for certain home or SOHO, it might be reasonable.
  • Wonder if there’d be a way to check against the ASN?

from: [“We have a message that says the sender is _____. Based solely on that, what should we do?”]

  • Make sure it’s a valid address: Email::Valid. If it’s an illegal address, trap the host.
  • Make sure the domain part has an A or MX record. This address, after all, is where bounces are supposed to be delivered. If there are no A or MX records, it can’t possibly play nice, so trap it. Email::Valid above can check this. greyscanner does the same, using the same. Wonder what the timeout values are? How many attempts at how many hosts for how long?
  • greyscanner has the nifty feature of looking for multiple from: domains. Why would a mail server you haven’t previously talked to start sending mail from: a bunch of different domains? Very clever. It’s perfectly reasonable that a new server could send to: different domains you manage. Although, it’s even reasonable that a new mailer be sending you multiple from: addresses in the same domain — a mailing list utilizing VERP, for instance. The following is obviously spam:
  • GREY||arborealcarbonium@roullet.com|john.smith@example.com|1169133522|1169147922|1169147922|1|0

  • Sender address verification? Stupid Verizon. Dumb idea. So if I’m at home and I telnet to a Verizon server’s SMTP port and say I’m bigbutts@obtuse.com (listen to the presentation at the top), then Verizon is going to go connect to foad.obtuse.com and try to probe bigbutts. Boy, that sounds funny. I’m pretty sure that’ll get their probe IP trapped there. Too easy to poison, and it’s pointless. So let’s not do this, because it’s Verizon stupid, which is just a shade stupider than AOL stupid and Comcast stupid. [As a side note, you MUST NOT refuse to accept a message based on the HELO parameter (RFC 2128, sec. 4.1.4). Bozos.]
  • Remember the null sender <> for bounces.
  • If the from: is from one of your domains, I’d trap it, assuming you have already whitelisted known other sender locations (your cell phone provider’s addresses), and users submit mail from a secured location (VPN) or use a different submission service.

to: [“We have a message for mailbox ________. Based solely on that, what should we do?”]

  • Whitelist. Handy thing would be to rotate passwords. Generate a little random userpart string (e.g., “aeifj13″) and have it automatically emailed to the helpdesk and admins. That’s your temporary whitelist password, good for a week maybe. If someone needs to send mail pronto to you, give them the password — tell them to send mail to aeifj13@example.com. Then have a process that scans and whitelists anyone sending email to the secret address. So if a prospect has a virus-infected server that got them on the CBL / SBL-XBL and now you’re blocking them, but they also have an RFP for $200,000, you’re just going to have to whitelist them.
  • If it’s a bogus address, you could trap the host. This is very effective. It would be even nicer if you could accept the mail, instead of bouncing it, to keep the spammer list dirty. It would be better still to have it act like a stupid user and visit every URL in every message, to confirm reading. Then, I suppose, spammers could insert hidden URL’s that a live human wouldn’t visit, just to let them know when their hand’s been tipped. But spamd works against pf tables, so you’d have to monkey around some other way. Redirect bogus mailers to a different MTA instance….?
  • Trap relay attempts. No sense letting them talk to your real server.
  • What about typos in the user part? Say they’re trying to email jack.andreeson@example.com and enter jack.andreson@example.com instead? Spellcheck it? You can block a lot by picking up senders sending to BS names, but would you want to block out a sales or press inquiry just because a prospect fat-fingered a real address? Might be reasonable to allow anything with a Levenshtein Edit Distance of 1 or 0 — only 1 deletion, insertion, or inversion to match, or already a match. This assumes you have a list of valid email accounts, which a corporate environment is easily done by dumping aliases, virtual mappings, LDAP email attributes, and the like.
  • Remember [] address literals.
  • < [abuse|postmaster]@…> might be troublesome, since those addresses are supposed to always exist, and they’re not infrequently configured to bypass any spam protection, so that anyone can forward problem reports to the mail maintainer of the domain. Sending a fake bounce (from <> to postmaster@) and then retrying half an hour later would be a surefire way to defeat a stock install. Don’t know how to work around that, having IP-based greylisting.
  • Plain postmaster (without any domain) is required to work, as well. RFC 2821, sec 4.5.1
  • greyscanner has a section using a regex against the to: , which is quite effective. Get a $6 domain and trap everyone sending to it (watch out for bounces from forged spam, though — not sure how others manage that). For instance, you get dsfijsfi23.org and post a few web pages with the address and then trap everyone sending to it. Well, some spammer sends mail with a header from: (“reverse-path“) of mesohorny@dsfijsfi23.org to nonexistent@example.com, and the example.com mail server bounces the message to you (“backscatter“), causing you to trap example.com’s sender IP, when they didn’t do anything wrong.

first: [“We have a connection from an IP address being made for the first time at _________ (3:13:46 a.m., for example). Based solely on the time of first contact, what should we do?”]

  • If someone has never talked to me before, and they’re sending me mail during my non-business hours (nights and weekends), I wouldn’t mind delaying them longer than the default passtime, since no one’s there to read it anyway. This wouldn’t work for global business getting a lot of time-critical mail in off-hours. But see first & to further along.

from & source ip: [“We have a connection from IP address __.__.__.__ with a sender of _______. Based solely on that, what should we do?”

  • SPF! Trap on hard-fail. Delay further on soft-fail. Do nothing on success. Perl Mail::SPF.

to & source ip: [“We have a connection from IP address __.__.__.__ trying to email mailbox ______. Based solely on this, what should we do?”]

  • The most I can think of here would be to use geolocation for certain recipients. The Asia-Pacific Sales Office mailbox might get a lot of new mail from China and Korea, while the Chicago Office Softball Team mailbox probably wouldn’t. There’s too much admin overhead, though, for a big company.

source ip & first: [“We have a new connection from IP address __.__.__.__ at time __:__:__. Based solely on these two things, what should we do?”]

  • And the most I can think of here would be geolocation, plus timezone offsets. Would this be normal operating times for the sender? That is, if it’s 3:00 a.m. in the UK, chances are we can extend the passtime on this from @co.uk for a while. Clunky. My 1and1.com account is hosted in Germany, though I’m in US/Central, so sender IP doesn’t necessarily correspond directly to the sender’s actual location. Can’t see actually doing this.

from & to & source ip: [“We have a connection from IP address __.__.__.__ trying to send mail to ______ from _____. Based on these three things, what should we do?”]

  • This is the ‘tuple’ of greylisting. If they try again after such and such a time, whitelist them.

source ip & from & first: [“IP address __.__.__.__ is sending mail from ______ for the first time at __:__:__. Based on these three things, what should we do?”]

  • Nothing I can think of.

source ip & to & first: [“IP address __.__.__.__ is sending mail to mailbox _______ for the first time at __:__:__. Based solely on these three things as a group, what should we do?”]

  • Well, using geolocation and timezone offsets (as above), see if it’s reasonable for the sender to be sending at this time, especially to: this mailbox. For instance, it’s 9:00 a.m. in Beijing, where this IP address ‘resolves’, and asia-sales or asia-support would be reasonable recipients. Otherwise, extend the greylist passtime. Another clunky one. But this is just brainstorming possibilities….. You actually read down this far? I’m flattered.

source ip & to & from & first: [“IP address __.__.__.__ is sending mail from ______ to _______ for the first time at __:__:__. Based solely on these four things as a group, what should we do?”]

  • Nothing I can think of.

from & to: [“We have a new connection trying to send mail from _______ to ________. Knowing only the from: and to:, what should we do?”]

  • And this would be personal whitelisting…. Clunky and cumbersome and way too much overhead. I can’t think of anything that would easily be automated as a rule.

from & first: [“We have a message from _______ being sent at __:__:__. Based solely on these two criteria, what should we do?”]

  • Nothing I can think of.

from & to & first: [“We have a message from _______ to ______ being sent at __:__:__. Based solely on these three criteria, what should we do?”]

  • Nothing requiring all three.

to & first: [“We have a message to _______ being sent at __:__:__. Based solely on these two criteria, what should we do?”]

  • This would be a more personalized version of the business hours scenario. We’re seeing mail for the very first time from a previously unknown IP address, sent to bob@example.com. Now, Bob works days (8:00a.m. to 5:00p.m., Monday-Friday). If the mail is sent at 6:00p.m. (Bob Time), he’s not going to see it for another fourteen hours anyway. Extend the greylisting time as far as reasonably possible so that if the sender is good, the message will be delivered before Bob comes to work in the morning. If the mail is sent at 7:30a.m., well just use the default 26 minutes, because he’ll be in shortly thereafter.

And that’s all I have at the moment…..

Sorry, the comment form is closed at this time.