Several years ago, in an attempt to improve Freenet’s performance, I came up with a new idea for how to route requests in Freenet.   Previously, Freenet would simply forward a request to a node that contains information similar to what was being sought.

This “next generation routing” algorithm was much smarter.  It would maintain detailed statistics about response times for different neighboring nodes, and forward a given request to whichever neighbor was most likely to provide the fastest response, based on their past performance.  The algorithm is described in-detail on the Freenet website.

We implemented this, in fact, we implemented a much more sophisticated version of this.  It would take into account the latency, throughput, reliability, and other relevant qualities of neighboring nodes, before making its prediction about the response time for each.

Unfortunately, we were never able to get it to work satisfactorily.  At the time, we did not know whether this was due to a fundamental flaw in the concept, or some bug or bugs in the implementation.  As has happened in the past, Freenet’s inherent complexity made diagnosing the problem akin to finding a needle in a haystack.  Isolating problems in P2P networks is especially hard because often the problem doesn’t manifest itself at the level of individual peers, but only at the level of the entire network. This means that the most common debugging technique, isolating the bug through a process of elimination, isn’t effective.

Anyway, this ambitious approach eventually gave way to a much simpler and better understood routing algorithm in Freenet 0.7, which we began in 2005, but we never did figure out whether the fundamental idea for NG routing was sound.

Well, it turns out that it was.  Last year my friend and occasional collaborator Oskar Sandberg (now employed by Google) wrote a paper called Decentralized Search with Random Costs which investigated this, and found that this technique was effective for efficiently routing in a small world network.

Perhaps some day we can revisit this in Freenet, although this time we’ll have to take a much more cautious approach to ensure we don’t wind up in the same “needle in a haystack” situation that we did last time.

According to Slashdot, Freenet has been nominated for the Sourceforge 2008 community awards as the project ‘Most Likely to be Shutdown By Government Agency’ (along with Tor, Truecrypt, and some others).

I guess its an honor, all publicity is good publicity as they say.  However, I don’t really consider getting shut-down by the US government to be a likely possibility unless there is a serious deterioration in the US political system, and fortunately I think the worst is behind us in terms of the abuses inspired by 9/11/01.

A few weeks ago I started hearing quite a bit of buzz about a new tool called the Freenet Messaging System or FMS.  I went to its website within Freenet (we call them “freesites”), and downloaded it.  Normally downloading and running code from an anonymous system isn’t a good idea, but Freenet makes it easy to digitally sign content, and the author, while anonymous, seems pretty honorable.

If you are familiar with Usenet, feel free to skip the rest of this paragraph.  Usenet is a discussion system that was very popular in the early days of the Internet.  To use Usenet you need a NNTP client, which are a bit like email clients,  and then you can post to newsgroups, forums on particular topics.  NNTP is the protocol used to access Usenet, just as HTTP is the protocol used to access the web, and SMTP is the protocol used to send emails.  Newsgroups have names like alt.humor.puns.  These days the easiest way to use Usenet is probably through Google Groups, which, among other things, provides a web-based interface to Usenet.  A number of dedicated NNTP clients still exist, one of the better ones is Thunderbird, which is primarily an email client created by the Mozilla Foundation, the creators of the Firefox web browser.

FMS evokes some nostalgia because it’s pretty-much Usenet over Freenet, allowing users to read messages posted to online forums, and also to post their own messages. The major difference is that because it is Freenet, everyone can remain anonymous.   Currently FMS  has about 30 boards on topics ranging from science, to religion, to discussion of Freenet and FMS itself. Here is what it looks like in Thunderbird:

Uploaded with plasq‘s Skitch!

FMS is not the first such messaging system in Freenet, another popular one is called Frost. Unfortunately, over the past few months Frost has been plagued by a familiar problem on the Internet, spamming. Unlike the Internet, Frost’s spammers are not motivated by financial gain, rather their goal seems to be pure mischief. Whatever their motivation, they’ve rendered Frost next to useless. Freenet’s anonymity makes it next to impossible to prevent spam using the same techniques that are conventionally employed on the Internet.

So how does FMS address this problem?  It’s pretty cool, but first I need to give a quick crash course in how Freenet works.

Freenet is basically a globally distributed “Map” or “Dictionary”.  You can insert data under “keys”, and then someone else, given the appropriate key, can retrieve that data.  Freenet keys are somewhat analogous to URLs on the world wide web, except unlike URLs, they don’t point to the physical location of the data.

Freenet has a few different types of keys, which impose various constraints on the data that can be inserted under them.  The type of key used by FMS is called a “Subspace-Signed Key” or SSK.  The SSK itself contains a cryptographic public key, and any data inserted under a SSK must be signed by the corresponding private key.  An SSK is used much like a file-system directory from which anyone can read, but only the “owner” of the SSK can write.  Websites in Freenet are typically inserted under an SSK so only the author can update them.

Typically, email is “push” – senders push email to your email inbox.  FMS takes the opposite approach, to receive messages from people you must “pull” it from their outbox.  Because user’s outboxes are SSKs, all of a user’s messages are automatically signed and their integrity is checked by Freenet.  With FMS, receiving messages from a particular person is a voluntary act – and the only result of spamming is to flood your own outbox.

So how do users decide whom they should pull from?  FMS creates a “web of trust” between users, where each user indicates how much they trust other users they’ve communicated with. If person A trusts person B, and person B trusts person C, then person A can trust person C – and so on. Here is a representation of the current web of trust in FMS:

If a user proves to be untrustworthy, by spamming for example, then users can change their trust rating for that user, and this will propagate through the trust web, so that others quickly stop pulling messages from the spammer’s outbox.

Ok, so how do you join the web of trust in the first place? FMS employs a clever mechanism based on the concept of “thinkcash” or CAPTCHAs. Every FMS user inserts these CAPTCHA tests, which can then be solved by a new user in order of that person to become minimally trusted by the existing user. Basically, you need to prove you are human. This is what this looks like in FMS:

Uploaded with plasq‘s Skitch!

FMS is not without its shortcomings.  It is likely that the necessity to pull from the outbox of every user will only scale to a limited point, however FMS is being actively developed, and it is likely that these problems can be overcome in due course.

If you would like to try FMS, go and install Freenet, then click on the link to the Freenet Message System that will appear on the front page, you can download it from there.  Pre-built versions of FMS are provided for Windows and Linux, however if you use a Mac you’ll need to compile it yourself (although this was pretty painless once I discovered I needed to add a -D USE_BUNDLED_SQLITE=ON to the cmake command).

You just gotta love Google, we’ve just announced a $18k donation by Google to Freenet. Its hard to believe that its been over 8 years since I started the Freenet project, and almost as hard to believe that we’ve managed to raise enough money in donations to maintain a full-time developer for most of that time (see here if you aren’t familiar with Freenet).

Freenet has also done some favors for Google over the years, for example Freenet was indirectly involved in Google’s hiring of my good friend, and former Freenet contributor, Adam Langley. Also, another Freenetter recently accepted a job with Google, but as he hasn’t started yet I won’t reveal his name.

In other Freenet-related news, on Monday we announced the first release-candidate for Freenet 0.7, the first major release of Freenet to include “darknet” functionality. Now, not only is it extremely difficult for someone (eg. your government) to watch what you are doing with Freenet, but it will be hard for them even to know that you are running Freenet at all.