With the mail server moved over, I was finally able to cancel the last of our Interland machines. I was happy to be include the following as my "reason for cancellation:

Because the level of service and support from Interland has been absolutely abysmal.

Despite admitting that a specific piece of hardware was "probably faulty", your service people charged me $50 to check the hardware on the machine, before admitting that that piece of hardware was not included in the test. Due to the constant crashes due to this faulty hardware, I have submitted many, many, many reboot requests (since, unlike your cheaper competitors, you don't have a way to do this automatically), I have, at times, waited over 45 minutes for these requests to be fulfilled. My sales person also stopped responding to requests for a transfer to a new machine.

Any level of service you guarantee has failed to be met, and I have since moved all my applications to a new provider where, miraculously, the exact same software has failed to have a single issue in over a month.

I have lost thousands of dollars of business due to your ineptitude, and have relayed all this to the hundreds of thousands of readers of our site. I really hope that this loses you some business, because you don't deserve it.

On the plus side, VoxRox has been awesome so far.

I've commented on Goats about our move off the crappy Interland machines, over to VoxRox. Last week and over the weekend I finally completed that move by getting our mail server onto the new machine. This is a sort of summary of what I did and my goals in doing it. None of it is particularly original, but I haven't seen anyone lay down a spam-mitigation-summary anywhere, so this is my reference for any future work I do, or problems I have.

The thing about setting up a mail server in this day and age is that it isn't as easy as just setting up a single application, and letting it run, at least not in the free (as in beer) software world. You need to install the actual mail server, but you also need to get a web interface up (because people demand that these days), which means you need to get IMAP working in addition to the older POP3. Most of all you absolutely need some sort of Spam solution running on the server.

Prior to the move, I was downloading to my machine well over 1500 spams every day. My goal in the past had been to try and limit the amount of spam that would reach me, but this meant that all those messages had to be processed through that machine, by both the mail server and SpamAssassin in order to be downloaded by me and then deleted. This seemed like a massive waste of resources, which was one of the motivating factors to rework our systems.

In re-thinking this for the new machine, I tried to approach it like network security, and also in a resource conservationist methodology. So:

1. Make sure that there are multiple layers of defenses against spam.
2. Try to layer those defenses so that the first ones trim the most fat while using the least resources, so that the later ones can use more resources to look at what's left.

Our server uses a qmail/vpopmail/courier-imap/squirrelmail/SpamAssassin set-up. I followed many of the steps at qmailrocks.org to set this up, but skipped his bit on setting up SpamAssassin and clamav. I had followed his guide exactly, with qmail-scanner, etc on my last mail server installation, and had found that letting qmail-scanner do all my anti-[virus|spam] work just used too many resources, and ground the server to a halt. It processes every incoming message, and 99% of what hits our servers is going to accounts that don't even exist. A fun side effect of having had a domain for upwards of 8 years.

So, within this set-up, the first step in stopping spam is to just not let it onto the machine. I used two methods for this, rblsmtpd and a "badrcptto" patch written by the author of (and found via) O'Reilly's qmail book. (The badrcptto patch didn't integrate easily with the qmailrocks mega-patch, so I had to manually integrate parts of it in afterwards.)

rblsmtpd is a "real-time" block list of known spam servers. It queries pseudo name-daemons as to whether the incoming message is coming from a known-bad host, and then denies the connection if it's from a suspected bad guy. The two servers I'm using are sbl.spamhaus.org and cbl.abuseat.org. I've always been a bit wary of these block lists, because it means that I never see the message, and if a "good guy" gets mistakenly classified as bad, it can be difficult to get de-listed. In the meantime, I wouldn't be getting any mail from them. While this is worrisome, I decided that I'll see how it holds up, and that in general, this was a far lesser evil then the unmanageable amount of spam I was getting. If I was running an ISP, I might not be able to make that decision, but for a small business, it just makes a lot of sense. Additionally, for the cost of a simple DNS lookup, I'm alleviating SpamAssassin from the need to run all it's tests on messages destined for the junk heap. Another factor in deciding to install this was reading Cory Doctorow's essay "What to Do About Spam?". He compares running SpamAssassin with something like Razor (more on that later) to a "suggestion mechanism" for spam checking. I decided to consider rblsmptd as "suggesting" that if you run an open relay, I don't necessarily want to share email with you.

badrcptto also blocks the message, but it blocks it for any message going "To" an address in a list I maintain. SInce massive amounts of our spam are going to the now-defunct email addresses we had set-up for each of the characters about 5 years (or more) ago, this was a simple way of stopping them from getting through.

My second layer of defense is SpamAssassin. I tried to be more thorough in setting it up than I had been in the past. Specifically, I decided to make sure that I was using one of the "Hash Sharing Systems", and including the network tests. SpamAssassin offers three different HSSs: Vipul's Razor, Pyzor, and the Distributed Checksum Clearinghouse (or DCC). These work by calculating a (highly probabilistically) unique numerical identifier (hash) for a message and then checking to see if it's in a central database of known good or known spam identifiers. The main variations on these products is which programming language they're in, whether the database stores both ham (known good) and spam message hashes, and how they decide which parts of a message to include in calculating the hash. While I could really increase the score for any given message by using all three, I decided to use just one for now, to conserve resources in the case of "lots of spam" (I would want to limit the number of network connections per message). For now, I'm using DCC. My two main reasons are that it checks for both spam and ham, which should increase it's "correctness", and it's written in C (as compared to perl (for Razor) and Python (for Pyzor), so it should be nominally more resource efficient.

The other network checks that SpamAssassin uses mirror rblsmtpd somewhat. However, in addition to just checking the relaying mail host, they also check the RLS in the spam, to see if they're known to be spammer sites, and ups the score if they are. I'm not convinced that these tests are really worth it, but I'll watch how they affect the scoring, and disable them if it seems unnecessary and if I need to speed things up.

My last layer of defense is my mail client (Eudora). It has it's own Junk scanner, that used to catch about half the stuff that SpamAssassin missed. It also has a tendency to pick up some ham just often enough that I need to watch what it catches, but not often enough that it's worth tweaking it's scoring.

So, what's the verdict?

So far, so good. It seem that either rblsmtp, badrcptto, or both are stopping enough spam that SpamAssassin is using a very manageable amount of resources. The spam I'm downloading overnight dropped from ~600-1000 messages to ~60. That's a quantity I can easily go through. Of those, SpamAssassin missed about 5, or just under 10%, instead of closer to 50% as was happening before. I could probably improve this even more by adding a couple more of the block list servers to my configuration, especially the lists that include suspected trojan-ized machines, but I'll wait a little bit before I see if that becomes necessary. I suspect that by blocking the spams at network time, I've cut the load enough that I could reinstate the qmail-scanner/clamav system into my mail set-up, but again I'm going to wait and see how this holds up for a month before I do that.

I'm a programmer, not a sysadmin. I really hate spending my time dealing with this stuff. But the downside of being "the" tech person for a business, is you have to do it. I'm hoping that by incorporating other dynamic systems (DNS block lists, DCC, and bayesian filtering (not discussed here)) into mine, I can keep my system relatively static and not have to futz with keeping up with the latest rules or whatever. In reality, I know the spammers are constantly figuring out new ways to waste my time, but hopefully I can fend them off long enough to be productive for just a little while.

Kevin Drum writes:

I wonder if the final phase of this strategy is behind Bush's Social Security posturing? Maybe the plan looks something like this:

1. Bush proposes private accounts for Social Security.
2. As expected, Democrats go to the mattresses in opposition. However, in an effort to demonstrate reasonableness they all agree — almost in passing — that of course they have nothing against encouraging savings, but that it should be done in addition to Social Security, not in place of it.
3. After pretending to give it a good try, Bush counts noses, realizes he can't win, and reluctantly agrees to settle for tax-free private accounts on top of Social Security, just like the ones Dems say they have nothing against. Of course, this will be the Republican version of tax-free private accounts — big, unrestricted ones that mostly help the well off — but by now the Dems can hardly oppose a compromise like this, can they?

I would not be at all surprised to find out that something like this is in the works. So far Bush has never shown a whole lot of caring about traditional conservative values (the same way he's never shown too much interest in "family/religious" values, when he's not just using the rhetoric), but he has shown a lot of interest in finding new ways for the wealthy to not have to pay taxes. This would appear to be a really good way of helping the wealthy to not pay taxes.

What I haven't seen anyone ever explain, however, (and part of this is probably because no one has ever presented a "real" plan, just lots of vague ideas) is what the difference between these "new" private/personal/"individual investment" accounts (or whatever the nom du jour is) is and IRAs, Roth IRAs, 401-Ks, or other existing retirement type plans.

Wampum writes, on the potential of Bush's two terms:

If we add rising interest rates, reduced consumption, lower economic growth, lower productivity growth, higher prices, decreased retirement security, and loss of confidence to a first term record of zero job growth, mounting national debt, and rising income inequality we have a domestic legacy that could last for ages.

It's discouraging to think about the amount of harm that (many|some) of us think has already been done over such a short period of time. I think it's unfeasible that 4 years could have had so much impact. This really makes me think that the "campaign to undermine the media" has been far too successful. Eric Alterman's What Liberal Media?" hasn't gotten much attention lately, but I think that what we're seeing isn't the result of four years, but the result of a few decades worth of coercing the media that people actually pay attention to (TV) into covering politics from a very one-sided perspective via (probably wrongly) blaming the media that people don't pay attention to (anything on paper) for being liberal. A lot of this has been under the radar just by being blamed on the sensationalization of media. We can protest and fight against Bush and modern (non-)conservatism all we want, but until something changes the dynamic of how information is presented to the masses, it won' make a difference. The figurehead is not the problem.

Technorati seems to be messing around with something they're calling "tags", as a system of web-based meta data. The main method that the tags are generated is form the category label in a blog post, which is probably the "correct" way of doing this. My goal, when something like this comes out, is to make sure that Goats is properly represented in the taxonomy. (At least until the system gets hijacked by spammers.)

In our current RSS feed, I went for very specific category tags, which is useful to the end user, but less useful to us in a system like this. For example, I use the category "goats comic" for a comic strip, instead of just "comic", which would get us better represented in the category. I have four options:

1. Ignore it
2. Use their alternate system of adding categories.
3. Change our tag to the "best" option they have widely used, where "best" is some combination of likely to be search on, and not so-cluttered that we'll get lost.
4. Start using multiple categories per entry.

Option one isn't really an option, or I wouldn't be thinking about it here. Option two would require me to add special tags to every post (e.g. Dumbrella, comics, webcomics) in order to get them where I think they "should" be. Option three is a decent short term fix, but as the categories we care about (if we care about them) change, it becomes less appealing. Long term, the fourth option is the only "real" option. I need to check the RSS spec to see what the rules about multiple categories are, and see about adding that. Also, right now the categories I use in the news system aren't very customizable, so that's probably something worth changing/fixing.

Their system really has to learn about stemming too, "comic" and "comics" should not be two separate lists.

Robert X. Cringely's 2005 predictions are up. Although no one will care, I write this for my own memory. Additionally, my two reasons for doing this are:

• Cringely is a relatively smart pundit. He has a broad enough understanding of enough disciplines ((tech/programming)/(business/economics)) that's it's worth paying attention to what he says.
• It is so much easier to critique what he says than to articulate my own thoughts.

So, with no further delay, my selected thoughts on some of Cringely's predictions:

1) Microsoft's entry into the anti-virus and anti-spyware businesses will be a disaster for users. This is based on everything I know about Microsoft, having watched the company for almost 28 years. They will make a big fanfare, spend a lot of marketing dollars, but in the end, the company simply won't be able to keep up with the demands of keeping virus signatures current, which isn't the real point of this gambit, anyway. There is so much to this story and so much that I could write that I think I'll do so next week, and just move on to the next prediction.

He's probably right. MS is perfectly capable of doing this if they wanted to, but it isn't in the company DNA. There's no money or glory in ti, the updates are a constant drain that they get no glory or fanfare for putting the effort into. If it was legally possible, it would be great to force MS to do this, then they might actually expend the resources to address some of the underlying vulnerabilities that viruses and spyware take advantage of. But frankly, these help drive the upgrade cycle for windows, so why should MS do anything more than cosmetic? (Also, if they make this part of the OS, then they're both admitting lot, and setting themselves up for liability type issues for admitting that they aren't taking care of those underlying issues. That will be a lot of fun to watch.)

3) Apple will take a big risk in 2005. This could be in the form of a major acquisition. With almost $6 billion in cash, Steve Jobs hinted to a group of employees not long ago that he might want to buy something big, though I am at a loss right now for what that might be. Or Apple might decide to throw some of that cash into the box along with new computers by deliberately losing some money on each unit in order to buy market share.

We might see that as early as next week with the rumored introduction of an el-cheapo Mac without a display. The price for that box is supposed to be $499, which would give customers a box with processor, disk, memory, and OS into which you plug your current display, keyboard, and mouse. Given that this sounds a lot like AMD's new Personal Internet Communicator, which will sell for $185, there is probably plenty of profit left for Apple in a $499 price. But what if they priced it at $399 or even $349? Now make it $249, where I calculate they'd be losing $100 per unit. At $100 per unit, how many little Macs could they sell if Jobs is willing to spend $1 billion? TEN MILLION and Apple suddenly becomes the world's number one PC company. Think of it as a non-mobile iPod with computing capability. Think of the music sales it could spawn. Think of the iPod sales it would hurt (zero, because of the lack of mobility). Think of the more expensive Mac sales it would hurt (zero, because a Mac loyalist would only be interested in using this box as an EXTRA computer they would otherwise not have bought). Think of the extra application sales it would generate and especially the OS upgrade sales, which alone could pay back that $100. Think of the impact it would have on Windows sales (minus 10 million units). And if it doesn't work, Steve will still have $5 billion in cash with no measurable negative impact on the company. I think he'll do it.

This is interesting, but it won't happen. Apple will not introduce a device without a clear cut purpose, and from past experience they know how fast that those cash reserves can go. This post will be up so close to finding out what they're introducing that it won't matter, but I expect that it will be some sort of home media appliance. Initially, it will primarily be expected to be used as an iTunes server. It will have some very rudimentary TV integration functionality, that will be poo-poo-ed right now, but will serve as a toe-hold to expand on later. (A wedge that they'll use to coerce media companies to allow us to move TV shows onto it, and then probably onto iPods, Apple can do this because they're still perceived as niche, so the media companies don't feel like they're giving up the homestead when they really are, and should be giving up more if they want to survive.) It will be administered not via a monitor, but via a custom application, much like the AirPort Express is administered via the Airport Admin Utility (or the XServe admin stuff). That app may be Mac only right now, but they'll have a windows version out within a few months.

4) The Recording Industries Association of America will continue to sue customers while their business slowly dissolves. The big threat here isn't file swapping, but affiliate programs like Apple's iTunes Affiliate Program that I am sure will be shortly copied by all the online music stores. These affiliate programs turn bloggers into shills and blogs into record stores, with the result that record company's last source of power -- marketing clout -- is taken away. This will take time, but it is the beginning of the end for old-style record companies.

This is genuinely interesting, I I hope that it's right.

5) WiMax will be a huge story by summer, but widespread adoption of the wireless networking technology will take at least another two years. In the meantime, though, nobody will make money on WiFi, but it will become ubiquitous anyway, especially with the arrival of 802.11n.

The business model will be the same as that for "coffee in hotel rooms", you won't want to be the one without it. In reality, it's going to be so cheap to offer in so many places (once it's ubiquitous, the costs are well distributed by definition.)

6) VoIP will continue to shatter the telephone industry with the arrival of WiFi phones, which might finally be the killer app for hotspots. Eventually, all the backbone suppliers will figure out that VoIP is their salvation and will either start their own VoIP companies or ally with big VoIP players.

I find this interesting, because I'm a geek, and I have zero interest in VOIP. If all I cared about was cheap communication, then I would be all over it, but there are two large events that spring to my NYC based mind from the last two years, that make me just not care: 9/11 and the 2003 blackout. In the first case, the phone was useless. The lines were flooded, and I could only communicate with people via email/goats. In the second case, email/goats/my internet connection were useless. But the phones have their own power, and part of the reason I pya Verizon is to have backup power supplies where necessary. SO (as long as I had an old-fashioned phone around) I could still use that, but not the internet. (Except of course, that I was out of town for the blackout, so none of that was actually an issue for me.) I like redundancy. I don't trust one company to provide it to me.

7) The trend of repurposing Linux-based consumer electronics devices through revised firmware will expand dramatically as people realize the cost-benefit advantage, AND nerds realize that they can sell reprogrammed WRT54GS stuff all over town and over the Internet.

From a business perspective, I might stop using Linux if these practices becomes too wide-spread. Personally, it sounds like some sort of virus/worm attack waiting to happen, and as a business I wouldn't want my ass sued for my compromised Linux device being an issue.

12) There is no evidence that Sun will change its current course, which is inexorably downward. I know Jonathan Schwartz thinks I'm crazy, but so far I am more right than he is, and hear no reason coming from him why that should change.

I really wonder about Sun. They could have a lot going for them, but they fail to articulate what that might be. They're the Apple for engineers (as compared to artists), but they're really failing to take advantage of that or do anything that's not also being don in the Linux space. They need innovation, big time. Alternatively, they need a major linux compromise to occur, that makes a lot of people want to have alternative systems out there with someone they can sue behind them. So I guess, looking at my previous reponses, I should really look into be ing a lawyer in the next few years.

13) While Intel thinks its 2004 course corrections will do the job, I just don't see much in the new product roadmap to get excited about. AMD will continue to grow at Intel's expense. And keep an eye on IBM's PowerPC introductions later in the year that should really give Intel fits, especially if they are accompanied by substantial OEM agreements.

I care little about this, but I thing that AMD has never proved themselves at doing much beyond chasing Intel. I wonder how much progress they'll show if they actually catch up.

Eh. That's it. Obviously he'll get some of these right, especially once hindsight can be applied to the ones that are sufficiently vague(which is why I ignored 8-11), but I don't think he'll break 75%.

Words of obviousness (mmm...ness-monster) from Ezra:

We don't need to turn out our base, we need to enlarge it. We need to make liberalism safe and attractive, and we need more Americans to demand more progressivism in their candidate. When Barry Goldwater turned the hell out of his base, he lost by one of the widest margins in American history. His base was too small. When Reagan turned out his base, he won repeatedly. Reagan made conservatism attractive, he made his base -- theoretically the same one Goldwater had -- huge. Which is why the whole idea of bases as monolithic groups is dead wrong, they're amorphous, they fluctuate in size. And they're not the answer. When liberalism is ascendant and its spokesperson appealing, our base will be huge. Until then, it'll never be large enough.