While I was getting a haircut today, I thought about the scalability problem of solutions like TEN and SMTP4All.

Digitally signing each message will reduce mail throughput significantly.  Some of the throughput loss will be offset by removal of spam which will reduce message traffic by as much as 90%.  Scalability problem remains still because mail traffic is not constant.

Add to this the cost of bi-directional authentication between sender-SMTP and receiver SMTP.  If messages per session is high (i.e. mail traffic between AOL and MSN), cost will be minimal.  But I suspect the average number of messages per session is pretty low, meaning near one message per session.

In "Another Whack at Spam", Tim Bray describes a solution similar to my Trusted E-mail Network (TEN) idea (also read "Fixing E-Mail") – via Dave whose blog I read before Tim's blog.  He arrived at the solution while talking about spam at Foo Camp with Jeremy Zawodny, Dave Sifry, and Doug Cutting.

He also thinks digital signing should be done by mail servers instead of users.  But his thinking differs from mine in how the trusted network of mail servers should be organized and the network's relationship with mail servers outside the network.  I believe the network should be backed by a business entity in charge of issuing and revoking certs to member mail servers, maintaining and providing trust rating information on mail servers and mail users, etc.

While I like loosely-coupled peer network as much as anyone, I believe PKI and responsiveness requirements call for a central authority.  Under Tim's solution, each mail servers are given too much room for misbehavior and removal of a rogue mail server takes too much time.  What I want is the ability to shutdown a rogue user or mail server within hours, not days.

Update #1 – 10/13/2003 11:15AM PST

Liz Lawley raised some key concerns that reminded me to fill in some missing pieces of the TEN model.

Open Source

Since there is nothing proprietary about TEN servers, there will be plenty of open source TEN-enabled mail servers and TEN-enabling patches for popular open source mail servers.  So TEN servers will be very affordable.

Private E-Mail Network

TEN servers should be able to use multiple PKI.  This means each TEN server will have multiple certs to sign e-mails with depending on the source and the destination of each e-mail.

If the mail exchange is completely within an organization, the message can be signed with a self-signed cert after checking to see if the sender meets the private TEN's trust rating requirement.  For mail sent outside the private TEN, a public TEN cert assigned to that TEN server should be used but only if the mail sender has sufficiently high TEN rating.

Free or Subsidized

A Private TEN does not have to charge fees.  For example, American universities can form a national private TEN, with each university or department running their own TEN server, that allows students and staffs from any of the member universities to exchange e-mail with each other without a fee.  Some universities could even sponsor some percentage of the fee for e-mails sent outside the University TEN.

Global Trusted E-Mail Network

While anyone can run a TEN server, not everyone will be able to get a Global TEN cert that enables e-mails to be trusted by anyone world-wide.  There are three ways to get a Global TEN cert:

1. Implicit Trust – you are trusted without doubt or reserve
2. Bonded Trust – you put up money to be trusted
3. Sponsored Trust – you are trusted by someone with a Global TEN cert.

Implicit and bonded trusts are obvious so I won't go into details about them unless someone asks.  Sponsored trust means a there is relationship between the sponsoring organization and the sponsored.  Each trusted mail sent from a sponsored organization affects both organizations if a complain is lodged against the mail.

For example, if a Stanford physics student sends out a mail with virus to someone, trust rating of Stanford Physics Department's TEN server and Stanford's TEN server will be degraded because the school sponsored the department.  If Stanford is a member of American University TEN, then the American University TEN's trust rating is degraded.

I have been thinking about using Sleepycat Software's Berkeley DB XML (BDBXML), the speedy open source XML database behind Kimbro Staken's XPath-happy prototype blogware, Syncato.  BDBXML license was confusing to me, particularly the word "redistribution", so I contacted them yesterday to find out if I could use it for free.

My situation is a common one in that I will have a single server driving several websites and web services, some of which will be commercial.  More servers might be added later, but still located at a single data center (ServerMatrix).  BDBXML license allows free use under this situation.  But the software that runs on my server(s) is being written at home which is in a different state.  Since my development machine is in California and my production server(s) are in Texas, I am in fact redistributing whenever I upload my software to the server(s), violating free use under BDBXML license.

Liz Pennell, Account Executive at Sleepycat Software, confirmed this but, recognizing that this might discourage developers from developing software based-on their new product, they graciously granted me free license.

Hi, Don,

I'm sorry for the delay in responding to you, but I wanted to discuss your case with some of my colleagues here.

In fact, if the development is at your home address and the hosting server is at a different postal address, the use you contemplate would typically be considered a redistribution event under the terms of our public license. As such, in the interests of applying the rules even-handedly across our user base, we'd need to specifically permit that use rather than just wink at it, if that makes sense.

However, I agree with you that this sort of use at least in spirit comports with the public license terms, as the software is only installed and performing useful work at a single physical site. It's also true that DB XML is still a young product, and we'd like to do what we can to promote its wider adoption.

Therefore, in this specific case given all the circumstances as detailed, I'd be willing to grant you a limited single-site exception to clause 3 of Sleepycat's public license, and permit your use, as described below, at no charge.

Since I knew my situation is far from unique, I asked Liz whether this exception was a general exception (?!?).  Her reply was:

I'm reluctant to commit to making this exception broadly into the future, but it's not something you need to keep secret. If an individual web site developer came to us at this point in the product's life, and described the kind of use that you're considering, we would be very likely to make the same call for the same reasons.

However, it wouldn't be prudent to assume that this will always be so, and we'd need to make the explicit judgment on a case-by-case basis every time.

There you have it, a Selective Early-Bird Special of sort.  If you are a developer in a similar siutation as me, you know what you should be doing right now.  Here is the e-mail address.

I have a feeling that Sleepycat's mailboxes will be full by Monday.  Sorry to wake you, Sleepycat.  Heehee.

Update – 2004/02/19:

Here is an update from Liz on the license which made a lot of sense to me:

Since nothing ever dies on the Net, I thought it was important to get back to you on the subject of qualification for free use of our software under Sleepycat's public license. My interpretation of the public license when you and I had our original e-mail exchange was actually too strict.

In fact, "redistribution" happens when a copy of Berkeley DB is installed *for actual use* on more than one physical site. In practice, this means that developers may build apps at multiple sites, or at a different site from where the application will be installed, because during the development cycle the application is not installed for actual use. If the application is installed for actual use at a single physical site, it qualifies for free use of Berkeley DB under the public license, regardless of where it may have been developed.

I spent a good part of today trying to setup mail server on my remote Red Hat 9 machine.  It came with sendmail which everyone agrees is a bloated sloth so I replaced it with Postfix using the instruction given in this document which probably represents the daily life of a Linux user.  No problem there.  I have heard that Maildrop and Courier IMAP combo was a good match for Postfix, so I looked for the RPMs via Red Hat's up2date.  Nothing.

I managed to find the source for Maildrop and tried to configure the build.  Nope.  cpp, C preprocessor, was not installed.  Again, not available via up2date.  Got it via rpmfind.com and installed OK.  Back to configuring Maildrop.  OK again.  Now build fails because g++ is missing.

Earlier, I had to install gcc-compat package because my server came without gcc installed.  Funny.  Apparently, that gcc-compat package didn't include g++.  Latest gcc-compat available on RH9 was an old version of gcc also.  So I considered whether to replace gcc-compat with the latest gcc package from gnu.org.  gcc-compat is an ominous name.  It implies that regular version of gcc is not compatible for some reason.  I punted and rigged Postfix to forward mail to my personal e-mail account with my ISP [1].  Urgh.  I didn't even get to Courier IMAP.

Red Hat distributions seems to have, in general, oddities that cause problems with many packages out there despite being the most popular Linux distribution.  Versions of packages bundled in Red Hat distributions or available for download via up2date are often much older than the latest.  Sometimes this can be dangerous.  For example, latest OpenSSL package available by up2date was 0.9.7a which has serious bus including the ASN.1 bug.  Red Hat 9, in particular, is a bundle of trouble because it uses NPTL library unlike other versions and distributions.

Stay away from Red Hat 9 everyone.  If possible, stay away from other versions as well.  Whatever performance NPTL brings, I can make it up more than enough by getting another cheap Linux box.

PS: Only bright spot in all this is the joy of using Webmin.  It really is great although documentation sucks.  There are so many great Webmin modules out there that one can only smile with satisfaction.

[1] One stupid Linux trick I learned was that in order for 'luser_relay' setting to work, you have to define 'local_recipient_maps' to be nothing.  Leaving it commented out is not the same.  What kind of talented idiots write sh*t like this?

I visit Carlos E. Perez's Manageability blog about once a week because he occasionally posts useful list of open source Java projects along with terse yet revealing comments.  I thought it might be useful to list the lists.

• Portal Servers (most recent)
• Workflow Engines
• Rule-based Engines
• Regular Expression Engines (benchmark)
• Two Fast XML Parsers

Carlos is a bit heavy on the Java cheerleading, but these gems are worth the visit.