Just thought I'd point out that that these forums do get spam-harvested. The email address that I created when haddie wanted to post that image for jon's bday has been added to spam lists. The only place that address ever occured was in this message [goats.com].

So, just thought I'd warn you all.

You could say that concerning just about every page on the web. That's why we removed all the email addresses from the site and added the contact forms.

I highly recommend that you do not post your email address here or anywhere else; you should definitely use the email cloaking for your profiles here. It may seem pretty simple, but bots aren't sophisticated enough to get past it.

Just thought I'd point out that that these forums do get spam-harvested.

They do, but relatively lightly. I get a goats related spam mail about once a month, compared to my main mail account which picks up a dozen or two a day. The little bit from here really doesn't bother me - my e-mail address(es) are so easily accessible over the net, that I'm pretty much screwed already.

Cheers,
Graham

Speaking of spam, and more specifically, combatting it, I am at present working on a collarborative architecture for filtering. It's P2P and works by group-clustering. Essentially, folks share signatures of spam-emails and peers tend to gravitate towards those folk who made "good" recommendations and have similar "hits" as themselves.

Because it is a collaborative recommendation type system primarily, it suffers from the pump-priming problem. The classification is done by people, which this means that the more folks that use it, people will tend to get less spam. (In future, if I get funding, there will be a case-based machine learning technique used for classification to reduce the work on the human and cases will be shared instead of simple signatures).

Still with me? Good!

Thing is, I've done the design part and now the fun coding part. I want to develop a plug-in for a mail user agent and get people to use it so I can have real-life data. Therefore, I'd be looking to see what mail client folks use (Outlook, Eudora, Mutt or whatever) and try to develop for whichever one most volunteers would use.

So could you nice folks tell me what mail client you use and whether you'd like to help. Also, if anyone would like to voice an opinion on the "goodness" or "badness" of my idea, criticism is always fun!

In Response to evilaltor (#7316):

Answer one : I use Eudora, and would be willing to help. But not, obviously, if it meant switching to outcast excuse.

Answer two : The idea has merit, but falls down basically by being in the general class of "stop spam by filtering" solutions. These are bad for reasons too obvious to go into here.

While I think your idea has the possibility of being a good system for a filter, I don't personally believe that filtering is the answer. Instead, we need an architecture which verifies the sender of all e-mail, at least to the extent that his ISP and ID held by that ISP can be uniquely and correctly identified. We can then pass the power to the receiver of the potential spam via, for instance, do not call registers, anti-spam legislation, or even mass boycotts of known spam-haven ISPs.

The problem with this solution is that it is a lot more complex and expensive to set up than Evilaltor's system, and needs much more widespread acceptance to be truly useful. There are people working on such a system now, and I fully expect that within the decade, spam will have been killed by their fine efforts. In the meantime, the system Evilaltor is proposing could be a very useful stop gap.

Cheers,
Graham

In Response to albionsoft (#7317):

It is true that filtering is not an ideal solution. Obviously, stopped spam at the root is ideal, but that has huge legal ramifications. Unfortunately, that means that correct provisions need to be put into judicial and regulatory systems. More unfortunately, legislation tends to be applied piecemeal (look at the plethora of legislation in the States alone) and tends to lag technology by quite a number of months, which is generations in the technology sector. That makes the technological solutions, or attempts at them ;-), the front runners for the moment.

As for the "name and shame" approach, the good folks at SpamHaus [spamhaus.org] are doing an excellent job, although they are being sued. In his response [spamhaus.org] to the suit, Steve Linford makes the very valid assertion that the recipient of the mail has the absolute right to do what they want with it (a point iterated by Graham above - do not call register, ISP reporting). The response is well worth a read, if you don't mind wasting some of your valuable Friday afternoon work time.

Uniquely identifying the senders of email is relatively straightforward (digital signatures, for example), but this needs to be coupled with legislation. Boycotting spammers' ISPs may not be a good solution - spamming is profitable and spammers can fund their own. Junking everything from particular ISPs is obviously ludicrous. Forcing ISPs to disclose information on clients is disturbing to me because of where it could lead.

Sorry for yakking on (it is Friday and I certainly don't want to do work!). I'm afraid I don't share your enthusiasm that spam will be done with inside a decade - look at spam snail mail. Orders of magnitude more expensive than e-spam, but still in use. I think that we need comprehensive, consistant legislation with the technological support to enforce it. I also think that this will probably never happen, or certainly not the "comprehensive, consistant legislation" part (see oxymoron [reference.com]). This convinces me that the filtering route is the best solution for now, and perhaps the future.

BTW - "outcast excuse" - I like

In Response to evilaltor (#7316):

Too late -- this already exists [cloudmark.com].

Of course, if your solution is free, then it might be worth pursuing.

In Response to jon (#7319):

cloudmark is currently free (i dont know what the subscription version gets you) - I've been using it for about a year I think (maybee more, can't remember) and its caught over 20,000 spam messages with maybee 10 false positives (which i just unblocked easily enough)

but I dont think that cloudmark does the kind of peer-grouping that evilaltor is describing. IMHO for that kind of grouping to work, you need a much larger critical mass along what cloudmark already has

In Response to evilaltor (#7316):

I've done the design part and now the fun coding part.
So you've done the first 90% of the job, and are now getting ready to do the other 90% of the job.

So could you nice folks tell me what mail client you use and whether you'd like to help
When I'm not using Eudora, I use POP3.

I'd *like* to help, but I'm not gonna. Among other things, I agree with others that filtering is the wrong solution. I also agree that the right solution raises severe privacy questions. But I disagree that spammers deserve privacy... :-)

In Response to evilaltor (#7318):

Part of my point is that anti-spam legislation is starting to come in anyway - some of it quite draconian. I think in this case, legislation is showing signs of leading technology. Of course, I think this is partly because the legislators do not believe that their laws will actually be used too much, so I expect that they will be rapidly scaled back when it becomes possible to enforce them properly, but that may lead to a reasonable balance.

I do believe that reporting to the ISP is actually a seriously useful tool. Most spammers go out of their way to exploit open relays, fake email addresses, etc. There's a reason for this - the ISPs are the ones who bear the brunt of spam costs and disgruntled customers. It is not in most ISPs interests to harbour spammers, quite the reverse.

Note that I did not suggest that ISPs should hand details of spammers to private individuals. That is definitely a step too far.

Finally, I don't believe that spam is going away, ever. I do think it can be contained. I get maybe three or four bits of junk snail mail a week, compared to maybe a dozen spam e-mails a day, and growing. Subtly different problems.

Cheers,
Graham

In Response to albionsoft (#7325):

Finally, I don't believe that spam is going away, ever. I do think it can be contained.

Well. I agree with you for a very liberal definition of SPAM.

I will never get my uncle to stop sending me those stupid jokes that he only thinks are funny because he's six beers to the wind by the time he checks his email.

But I firmly believe that if a system is put in place to legally and financially (not necessarily PERSONALLY) identify the actual sender of each and every email message, and to impose real costs against those senders that send out spam, then the problem of spam as we know it today will go away.

In other words, if it becomes too expensive for spammers to send spam and they know they can't fake who they are, they'll go back to sending junk paper mail and being telemarketers, and buying pop-up ads on web sites.

In Response to zamphir (#7327):

In other words, if it becomes too expensive for spammers to send spam and they know they can't fake who they are, they'll go back to sending junk paper mail and being telemarketers, and buying pop-up ads on web sites.

Problem is that I don't think that will ever happen. The cost of sending e-mail will never reach, let alone exceed, the cost of sending paper post. And what is deemed "reasonable" behaviour or an "honest mistake" is going to be broad enough that we'll always have *some* spam. It's the sheer volume that is going to vanish, along with the most worrying forms (like all the porn and get rich quick schemes).

In short, I think there's always going to be loopholes that let through some spam, but I think the bulk of the problem can and will be sorted.

Cheers,
Graham

In Response to zamphir (#7327):

<pedant>SPAM is a canned meat; spam is unwanted email. (see subject of topic)</pedant>

I think that the relative anonymity of email is one of its bonuses. Unfortuneately, that does mean that it can be abused.

In Response to albionsoft (#7329):

The cost of sending e-mail will never reach, let alone exceed, the cost of sending paper post

I'm not saying that it should cost them anything to actually SEND the email. I'm just saying that they should be liable for costs AFTER the fact. If that is made true, and well-and-fiercely enforced, it will turn into a perceived cost of sending email.

I saw a relatively good idea in an IBM technical journal the other month that suggested that every email sent would require essentially putting a small amount of money into escrow, for refund as seen fit by the reciepient of the email. The problem with the way the model was presented is that there was no workable way to handle difference of opinions. The payments went to the recipient automatically - and if you didn't want to give them that money, you didn't send them any email.

Also, I don't like the notion of pay-up-front charges on email. Pay-after-the-fact is a lot better. It's more manageable, and more commercializable. It's still subject to abuse, like everything else, but it's easier to control the abuse - like the difference between using a credit card and a debit card.

But as I said before, it requires that every email sender be identifiable in some manner. Liable for their actions, as it were. It's only a breech of privacy because generally we don't allow anonymous entities to conduct business. If we did, those anonymous entities could be liable and the real people behind those entities would be liable - but the real people wouldn't be IDENTIFIED as being liable. On paper, it's only the entities that pay the fines.

In Response to jon (#7319):

I know about these good folks ;-)

Initially I was quite disheartened by it, but there are some interesting differences between Cloudmark's SpamNet and my one.

The key point from my course (Networks and Distributed Systems) perspective is the fact that my one is totally decentralised P2P. Therefore, there are no expensive servers to maintain etc. (probably like your favourite file-sharing apps). This means that it is cheaper to do stuff, and you can configure your own node to participate as it wants (for example, how it ranks "similar" folk, or if you trust people more if they present a valid digital signature with their email).

Another one is the evolvability of my proposed solution. The protocols I've designed can be used to form C/S, P2P and other topologies based on the interpretation of them, so there are a number of directions for future development. This is particularly useful for "light peers" such as folk using PDA's to access email.

One major shortcoming with SpamNet is that it requires an extra round trip per email, that is, you receive a mail, compute a fingerprint, send that off and get a "yae" or "nae" response. That may be fine for always-on desktop email, but PDA useage will make this a prohibitive bottleneck, as the price per email will triple. Of course, long-term wireless connectivity and ubiquitous computing will put paid to that, but probably not in the forseeable decade.

The most interesting future development (in my eyes, epsecially as I'm looking for funding in it) would be the extension of the network to include collaborative machine learning techniques. Machine learning has already been used successfully on single node mail classification, so this is the obvious next step. It also means I get to use big words like "anthropomorphisation" and abuse college bandwidth for a few more years whilst getting paid to do it!

Of course, these are all very interesting to academics (I hope!), but what about the bottom line for Joe average? Well, P2P is much more scalable and cheaper (compare computational power of SETI@home to a supercomputer), so that will drive the cost down. By decoupling the user-agent and the underlying protocols (like the FastTrack P2P Engine and the myriad of Gnutella clients), there is a business model for free and non-free networks to evolve.

In Response to zamphir (#7322):

So you've done the first 90% of the job, and are now getting ready to do the other 90% of the job.
Preaching to the choir there, Zamphir. I still gotta write it all up too, which is the third 90% of the work.

I'd *like* to help, but I'm not gonna.
I know the feeling. To be honest, I'd be shocked if anyone here actually did want to use it <strike>'cos you're all bastards </strike> 'cos I wouldn't trust stuff that's in dev that people I don't know write. What I was hoping for (and got) was an interesting and impartial evaluation/discussion from intelligent people who's opinions I respect. Can't get that 'round my lab - everyone's wrapped up in their own thoughts and will agree just to make life easier.

In Response to zamphir (#7332):

The problem with the way the model was presented is that there was no workable way to handle difference of opinions.
  I think another problem with that would be similar to the Micropayments discussion - the actual cost of the payments and transactions infrastucture might cause the system to implode.
   
  ...it requires that every email sender be identifiable in some manner.
  This has serious ramifications for privacy, and not just in the "virtual" world. Doing something like that for electronic communication must have parallels in the non-electronic sphere. Every person would have to be explicitly identified in every case, on the off-chance they might be trying to conduct business.
   
  By extension, people would have to be explicitly identified in any scenario where they may be trying to consuct business. This would be in all human interaction. The possibility of wining and dining clients would mean that all bars/restaurants require logs of each and everyone on their premises, similarly conversations on public transport can be of a business nature. Maintaining transparent logs of all converstation and making them available for scrutiny is without a doubt, improbably difficult, not to mention offensively and illegally intrusive.
   
 

In Response to evilaltor (#7335):

This has serious ramifications for privacy, and not just in the "virtual" world.

A) I disagree. What I am specifically discussing is a system whereby I can register an alias, like "Zamphir", and do business as that alias, thrugh a trusted third party. The legal connection between "Zamphir" and me is maintained, but maintained at a much higher degree of security than it is today. Sealed by the court, as it were, completely unavailable to anyone except under very strict provisions.

B) Your attempts to extend my model into the real world reducto (reductio?) ad absurdum fail because I am specifically talking only about email.

The model works kinda like this:
I create an anonymous entity with a financial institution. I provide the financial institution with access to sufficient fund streams to allow them to charge me as appropriate. The financial institution is legally required to keep my *actual* identity as private as they can - and are only allowed to disclose my real identity in certain very specific circumstances.

I then register that anonymous entity with an ISP and get an email address. The ISP is able to charge back to the financial information for any fines levied against me for my improper use of email.

There are two reasons the US government won't ever allow this sort of thing: Taxes and Terrorism.

But I can dream...

And again, spammers don't deserve privacy. Neither do child molesters.

In Response to zamphir (#7336):

reducto (reductio?) ad absurdum
  Are you trying to hex me? Damn, I knew I should have paid more attention in DA meetings.
   
  Ok, allowing that you can do this at a cost that is not prohibitive and a system that is secure enough, I don't believe it in itself will solve the problem. It provides a non-repudiation service, but that service is just that - it allows for undeniable proof that the person under that alias is that person [1]. So it solves nothing by itself, which still leaves the very difficult problems of A)defining what constitutes proper and improper use of email and B)categorising it such a degree of accuracy that there are no false negatives.
   
  The proof by extension thing I was trying fell flat on its face. I accept that. It's not an argument of mine, but one I heard from a friend of mine in law. Something along the lines of it being applied in all frames of reference. Perhaps someone else can express it better than I.
   
  [1] On another note, a non-repudiation service like this scares me. It's like using biometrics for security. Sure, my retinal scan is unique and therefore secure. But once it is possible to forge it (and it will be), I am screwed. It's not like I can change my eyeball. That leaves me open to extraordinary abuse by the powers that be as they stand today with the lack of transparency in the system. Ok, so aliases etc can be changed, but passwords etc can be cracked and again until I change it, I cannot repudiate any allegations against me.

In Response to evilaltor (#7335):

The problem with the way the model was presented is that there was no workable way to handle difference of opinions.

I think another problem with that would be similar to the Micropayments discussion - the actual cost of the payments and transactions infrastucture might cause the system to implode.


There is a scheme by which you could make it work temporarily, but it would be a victim of its own success (and would, hence, eventually implode). Just for shits and giggles, here's the idea:

The money placed in escrow does not go to the email recipient no matter what. If the email is not "accepted", then the money goes to the infrastructure. This is vital for two reasons:

First, it is the only way to properly align the motives of the email recipient. You don't want to give the recipient an economic incentive to falsely reject a message. Lacking such incentive, his goal in using the system is the same as the system's goal -- fight off spam.

Second, it's the only way to pay for infrastructure. If an email is accepted, then the escrow money should be refunded to the sender -- in other words, there's no source of funds for the transaction. But the transaction still costs money. Something has to subsidize all the transaction for "good" email, and the only option (other than a general tax on email) is to use the penalties from "bad" email. So, if a spammer sends one reject message and is charged (say) $1, that covers the cost of "his" transaction (the $1 penalty itself) plus the cost of some number of transactions for accepted emails.

The fatal flaw in this approach should now be strikingly obvious. It only works if the ratio of bad emails to total emails exceeds the ratio of transaction cost to penalty cost. But it also can only be said to work if it replaces the need for spam filters, meaning that it must work by reducing the amount of spam sent. As it makes spam economically unfeasible, it drives spammers out of the market, making itself economically unfeasible. Once it collapses, the spammers come back out of hiding.

Any successful anti-spam model has to either (a) have zero overhead for legitimate emails (which is impossible with an escrow-on-send approach) or (b) impose some form of email tax.

So, while I do think the anonymity issues could be resolved, that doesn't matter.

In Response to mea37 (#7340):

Any successful anti-spam model has to either (a) have zero overhead for legitimate emails (which is impossible with an escrow-on-send approach) or (b) impose some form of email tax.

I disagree. Zero overhead for legitimate emails can be accomplished. Escrow funds go into interest bearing accounts, and the interest is granted to the infrastructure organization.

As long as your balance of unaccepted emails is smaller than the amount in your escrow, you never have to pay another dollar.

The infrastructure organization is responsible for how it invests the escrow money - and thus it's economic interest is to preserve the escrow fund and prevent rejected emails. Because it's only when an email is rejected that any principle leaves the escrow fund.

Which means that the infrastructure organization is an insurance company - with all the difficulties that brings. However, it makes the legislation and regulation part to get this set up a lot easier.

Anyone wanna buy spam insurance?

In Response to zamphir (#7341):

I disagree. Zero overhead for legitimate emails can be accomplished. Escrow funds go into interest bearing accounts, and the interest is granted to the infrastructure organization.

Ok, but for the system to sustain itself, it has to work when only legitimate emails are sent, so that means that on average the interest derived from holding the funds for an accepted email would have to exceed the cost of the transaction.

Suppose the escrow amount were $10 per email and the holding period were a month. Even with such absurd parameters, you would need a 12% yield to cover a $0.10 transaction cost. Even in an exceptional economy, I don't think you'll find better than 6% on a suitable investment.

Ultimately, what you're describing is equivalent to taxing all email, because the sender loses the ability to invest the escrow funds while they're being held. If the money gained by the infrastructure is significant, then the money lost by the senders is also significant.

Looking at that last statement, you might think "Ah, but the gains for the infrastructure are multiplied by the huge user base." Close, but not quite. Suppose you have 10,000,000 users. Your escrow amount is $10/email, and on an average day you process 5 emails/user. So each user stakes $50 for the day's email. Assume your user base is wealthy enough to accept that cost, knowing they'll get it back when the message has been accepted. So you get to hold $500,000,000 and invest it.

Now suppose you hold funds for 1 month, and again your senders are willing to put up with this cost. Once you're up and running, you can expect to hold $15,000,000,000 in escrow.

Someone with that kind of money has a lot of investment options, but you still have to watch your risk. If the system craps out, you're going to end up owing substantial amounts of money to each of your 10 million users.

Suppose you find a suitable 10% investment. Each year you have $1.5 billion to cover infrastructure. Of course, you have to process 18,250,000,000 transactions, each involving two authenticated transfers of funds. If such a transaction cost more than 8.219 cents (basically 4 center per transfer, ignoring any fixed costs of operating the system), the system will fail. It will also fail if the economy is weak for a few years or if your users aren't each willing to give up use of $1500.

In Response to zamphir (#7341):

Because it's only when an email is rejected that any principle leaves the escrow fund.

Missed this on a first read: How do you figure? If I send an email that isn't rejected, I get my money back. That's my incentive to send mail that isn't spam. I'd argue that only when an email isn't rejected does principle leave the fund.

In Response to zamphir (#7332):

I saw a relatively good idea in an IBM technical journal the other month that suggested that every email sent would require essentially putting a small amount of money into escrow, for refund as seen fit by the reciepient of the email.

Another problem: How would the system deal with mailing lists? I see three main options. There are variations, but they all suffer the same basic flaws as these three:

1) Mailing lists are exempt. Provides an obvious loophole for spammers, so no good. Any attempt to close the loophole would require some kind of central administration/registration of mailing lists, which would be a Bad Thing[tm].

2) If the list has 100 users, the message originator is responsible for 100 escrow payments (the fate of each decided by one of the recipients). Again, this requires special treatment for the list maintainers, and it also provides some interesting opportunities for them to commit fraud. Also, the system will inevitably be used as a weapon whereby I send an unpopular idea in a message to the list, 5% of the recipients accept what I've written, so I lose 95% of my escrow fees.

3) The originator is responsible for 1 message; the list maintainer is responsible for 100. This is the most natural approach and requires no special treatment of mailing lists, but it obviously misaligns responsiblity for spam. If I can pay 1 escrow fee to hit 100 sets of eyes, maybe that's a low enough cost that I do it. Sure, the list admin gets pissed and boots my ass, but then it breaks down into the same game spammers play with ISP's today. Except that with mailing lists actually paying fines for the leeching spammers, eventually all mailing lists fold in disgust.

In Response to Dynedain (#7330):

SPAM is a canned meat

That's an awfully generous interpretation.

In Response to mea37 (#7344):

Missed this on a first read: How do you figure? If I send an email that isn't rejected, I get my money back. That's my incentive to send mail that isn't spam. I'd argue that only when an email isn't rejected does principle leave the fund.

Why would you want your money back, when you would then have to put the same money back in?

It's like this: I put $100 into escrow. I am nominally charge d say $0.10 for each email I send out. As long as I never have more than 1000 emails that are neither accepted nor rejected, I do not have to put in any more money. Ever. Any email that is accepted causes me to be refunded that $0.10 back into my escrow fund. Any email that is rejected causes me to lose that $0.10. The $0.10 is not actually transferred from my escrow fund to someone else's unless an email is rejected. It is merely 'locked'. This is a significantly smaller transaction load and thus transaction fee.

Your incentive to send email that isn't spam is still the same - you don't want to deplete your escrow fund and have to actually put out MORE money.

I haven't spent a lot of time thinking about your mailing list argument, but my feeling is that it is the list moderator, and not the individual subscribers, who get to chargably accept or reject an email message. The moderator then has the option of rejecting the message as if it went to one person, or to all subscribers.

I just hope J&P aren't taking notes for Goats:The Forums 3.0....