Freenet Communication Primitives: Part 2, Service Discovery and Communication

Do, 08/20/2015 - 19:42 — Draketo

Basic building blocks for communication in Freenet.

This is a guide to using Freenet as backend for communication solutions - suitable for anything from filesharing over chat up to decentrally hosted game content like level-data. It uses the Python interface to Freenet for its examples.

Mirror,
Freenet Project,
License: GPL.

This guide consists of several installments: Part 1 is about exchanging data, Part 2 is about confidential communication and finding people and services without drowning in spam and Part 3 ties it all together by harnessing existing plugins which already include all the hard work which distinguishes a quick hack from a real-world system (this is currently a work in progress, implemented in babcom_cli which provides real-world usable functionality).

Note: You need the current release of pyFreenet for the examples in this article (0.3.2). Get it from PyPI:

# with setuptools
easy_install --user --egg pyFreenet==0.4.0
# or pip
pip install --user --egg pyFreenet==0.4.0

This is part 2: Service Discovery and Communication. It shows how to find new people, build secure communication channels and create community forums. Back when I contributed to Gnutella, this was the holy grail of many p2p researchers (I still remember the service discovery papers). Here we’ll build it in 300 lines of Python.

Welcome to Freenet, where no one can watch you read!

USK: The Updatable Subspace Key
KSK: Upload a file to a password
KSK queue: Share files by uploading to a password
KSK queue with CAPTCHA
Convenience: KSK queue with CAPTCHA via USK to reference a USK
Many-to-many: KSK->CAPTCHA->USK->USK which is linked in the original USK

USK: The Updatable Subspace Key

USKs allow uploading increasing versions of a website into Freenet. Like numbered uploads from the previous article they simply add a number to site, but they automate upload and discovery of new versions in roughly constant time (using Date Hints and automatic checking for new versions), and they allow accessing a site as <key>/<name>/<minimal version>/ (never understimate the impact of convenience!).

With this, we only need a single link to provide an arbitrary number of files, and it is easy and fast to always get the most current version of a site. This is the ideal way to share a website in Freenet. Let’s do it practically.

import os
import tempfile

import fcp
n = fcp.node.FCPNode()
# we create a key again, but this time with a name: The folder of the
# site: We will upload it as a container.
public, private = n.genkey()
# now we create a directory
tempdir = tempfile.mkdtemp(prefix="freesite-")
with open(os.path.join(tempdir, "index.html"), "w") as f:
    f.write('''<html>
    <head>
    <link rel="stylesheet" type="text/css" href="style.css">
    <title>First Site!</title></head>
    <body>Hello World!</body></html>''')

with open(os.path.join(tempdir, "style.css"), "w") as f:
    f.write('body {color: red}\n')

uri = n.putdir(uri=private, dir=tempdir, name="hello",
               filebyfile=True, allatonce=True, globalqueue=True,
               usk=True)
print uri
n.shutdown()

But we still need to first share the public key, so we cannot just tell someone where to upload the files so we see them. Though if we were to share the private key, then someone else could upload there and we would see it in the public key. We could not be sure who uploaded there, but at least we would get the files. Maybe we could even derive both keys from a single value… and naturally we can. This is called a KSK (old description).

KSK: Upload a file to a password

KSKs allow uploading a file to a pre-determined password. The file will only be detectable for those who know the password, so we have effortless, invisible, password protected files.

import fcp
import uuid # avoid spamming the global namespace

n = fcp.node.FCPNode()
_uuid = str(uuid.uuid1())
key = "KSK@" + _uuid
n.put(uri=key, data="Hello World!",
      Global=True, persistence="forever",
      realtime=True, priority=1)
print key
print n.get(key)[1]
n.shutdown()

Note: We’re now writing a communication protocol, so we’ll always use realtime mode. Be aware, though, that realtime is rate limited. If you use it for large amounts of data, other nodes will slow down your requests to preserve quick reaction of the realtime queue for all (other) Freenet users.

Note: Global=True and

persistence="forever"

allows telling Freenet to upload some data and then shutting down the script. Use async=True and waituntilsent=True to just start the upload. When the function returns you can safely exit from the script and let Freenet upload the file in the background - if necessary it will even keep uploading over restarts. And yes, Capitcalized Global looks crazy. For pyFreenet that choice is sane (though not beautiful), because Global gets used directly as parameter in the Freenet Client Protocol (FCP). This is the case for many of the function arguments. In putdir() there’s a globalqueue parameter which also sets persistence. That should become part of the put() API, but isn’t yet. There are lots of places where the pyFreenet is sane, but not beautiful. It seems like that’s its secret how it could keep working from 2008 till 2014 with almost no maintenance

For our purposes the main feature of KSKs is that we can tell someone to upload to an arbitrary phrase and then download that.

If we add a number, we can even hand out a password to multiple people and tell them to just upload to the first unused version. This is called the KSK queue.

KSK queue: Share files by uploading to a password

The KSK queue used to be the mechanism of choice to find new posts in forums, until spammers proved that real anonymity means total freedom to spam: they burned down the Frost Forum System. But we’ll build this, since it provides a basic building block for the spam-resistant system used in Freenet today.

Let’s just do it in code (descriptions are in the comments):

import fcp
import uuid # avoid spamming the global namespace

n = fcp.node.FCPNode()
_uuid = str(uuid.uuid1())
print "Hey, this is the password:", _uuid
# someone else used it before us
for number in range(2):
    key = "KSK@" + _uuid + "-" + str(number)
    n.put(uri=key, data="Hello World!", 
          Global=True, persistence="forever",
          realtime=True, priority=1,
          timeout=360) # 6 minutes
# we test for a free slot
for number in range(4):
  key = "KSK@" + _uuid + "-" + str(number)
  try:
    n.get(key, 
          realtime=True, priority=1, 
          timeout=60)
  except fcp.node.FCPNodeTimeout:
    break
# and write there
n.put(uri=key, data="Hello World!",
      Global=True, persistence="forever",
      realtime=True, priority=1,
      timeout=360) # 6 minutes
print key
print n.get(key)[1]
n.shutdown()

Note that currently a colliding put – uploading where someone else uploaded before – simply stalls forever instead of failing. This is a bug in pyFreenet. We work around it by giving an explicit timeout.

But it’s clear how this can be spammed.

And it might already become obvious how this can be avoided.

KSK queue with CAPTCHA

Let’s assume I do not tell you a password. Instead I tell you where to find a riddle. The solution to that riddle is the password. Now only those who are able to solve riddles can upload there. And each riddle can be used only once. This restricts automated spamming, because it requires an activity of which we hope that only humans can do it reliably.

In the clearweb this is known as CAPTCHA. For the examples in this guide a plain text version is much easier.

import fcp
import uuid # avoid spamming the global namespace

n = fcp.node.FCPNode()
_uuid = str(uuid.uuid1())
_uuid2 = str(uuid.uuid1())
riddlekey = "KSK@" + _uuid
riddle =  """
What goes on four legs in the morning,                          
two legs at noon, and three legs in the                         
evening?
A <answer>
"""
# The ancient riddle of the sphinx
n.put(uri=riddlekey, data="""To reach me, answer this riddle.

%s

Upload your file to %s-<answer>
""" % (riddle, _uuid2),
      Global=True, persistence="forever",
      realtime=True, priority=1)

print n.get(riddlekey, realtime=True, priority=1)[1]
answer = "human"
print "answer:", answer
answerkey = "KSK@" + _uuid2 + "-%s" % answer

n.put(uri=answerkey, data="Hey, it's me!",
      Global=True, persistence="forever",
      realtime=True, priority=1)

print n.get(answerkey, realtime=True, priority=1)[1]
n.shutdown()

Now we have fully decentralized, spam-resistant, anonymous communication.

Let me repeat that: fully decentralized, spam-resistant, anonymous communication.

The need to solve a riddle everytime we want to write is not really convenient, but it provides the core of the feature we need. Everything we now add just makes this more convenient and makes it scale for many-to-many communication.

(originally I wanted to use the Hobbit riddles for this, but I switched to the sphinx riddle to avoid the swamp of multinational (and especially german) quoting restrictions)

Convenience: KSK queue with CAPTCHA via USK to reference a USK

The first step to improve this is getting rid of the requirement to solve a riddle every single time we write to a person. The second is to automatically update the list of riddles.

For the first, we simply upload a public USK key instead of the message. That gives a potentially constant stream of messages.

For the second, we upload the riddles to a USK instead of to a KSK. We pass out this USK instead of a password. Let’s realize this.

To make this easier, let’s use names. Alice wants to contact Bob. Bob gave her his USK. The answer-uuid we’ll call namespace.

import fcp
import uuid # avoid spamming the global namespace
import time # to check the timing

tstart = time.time()
def elapsed_time():
    return time.time() - tstart


n = fcp.node.FCPNode()

bob_public, bob_private = n.genkey(usk=True, name="riddles")
alice_to_bob_public, alice_to_bob_private = n.genkey(usk=True, name="messages")
namespace_bob = str(uuid.uuid1())
riddle =  """
What goes on four legs in the morning,                          
two legs at noon, and three legs in the                         
evening?
A <answer>
"""
print "prepared:", elapsed_time()
# Bob uploads the ancient riddle of the sphinx
put_riddle = n.put(uri=bob_private,
                   data="""To reach me, answer this riddle.

%s

Upload your key to %s-<answer>
""" % (riddle, namespace_bob),
                   Global=True, persistence="forever",
                   realtime=True, priority=1, async=True,
                   IgnoreUSKDatehints="true") # speed hack for USKs.

riddlekey = bob_public
print "riddlekey:", riddlekey
print "time:", elapsed_time()
# Bob shares the riddlekey. We're set up.

# Alice can insert the message before telling Bob about it.
put_first_message = n.put(uri=alice_to_bob_private,
                          data="Hey Bob, it's me, Alice!",
                          Global=True, persistence="forever",
                          realtime=True, priority=1, async=True,
                          IgnoreUSKDatehints="true")

print "riddle:", n.get(riddlekey, realtime=True, priority=1, followRedirect=True)[1]
print "time:", elapsed_time()

answer = "human"
print "answer:", answer
answerkey = "KSK@" + namespace_bob + "-%s" % answer
put_answer = n.put(uri=answerkey, data=alice_to_bob_public,
                   Global=True, persistence="forever",
                   realtime=True, priority=1, async=True)

print ":", elapsed_time()
# Bob gets the messagekey and uses it to retrieve the message from Alice

# Due to details in the insert process (i.e. ensuring that the file is
# accessible), the upload does not need to be completed for Bob to be
# able to get it. We just try to get it.
messagekey_alice_to_bob = n.get(answerkey, realtime=True, priority=1)[1]

print "message:", n.get(uri=messagekey_alice_to_bob, realtime=True, priority=1,
                        followRedirect=True, # get the new version
                        )[1]

print "time:", elapsed_time()
# that's it. Now Alice can upload further messages which Bob will see.

# Bob starts listening for a more recent message. Note that this does
# not guarantee that he will see all messages.
def next_usk_version(uri):
    elements = uri.split("/")
    elements[2] = str(abs(int(elements[2])) + 1)
    # USK@.../name/N+1/...
    return "/".join(elements)

next_message_from_alice = n.get(
    uri=next_usk_version(messagekey_alice_to_bob),
    realtime=True, priority=1, async=True,
    followRedirect=True) # get the new version

print "time:", elapsed_time()
# Alice uploads the next version.
put_second_message = n.put(uri=next_usk_version(alice_to_bob_private),
                           data="Me again!",
                           Global=True, persistence="forever",
                           realtime=True, priority=1,
                           IgnoreUSKDatehints="true",
                           async=True)

# Bob sees it.
print "second message:", next_message_from_alice.wait()[1]
print "time:", elapsed_time()

print "waiting for inserts to finish"
put_riddle.wait()
put_answer.wait()
put_first_message.wait()
put_second_message.wait()
print "time:", elapsed_time()

n.shutdown()

From start to end this takes less than 2 minutes minutes, and now Alice can send Bob messages with roughly one minute delay.

So now we set up a convenient communication channel. Since Alice already knows Bobs key, Bob could simply publish a bob-to-alice public key there, and if both publish GnuPG keys, these keys can be hidden from others: Upload not the plain key, but encrypt the key to Bob, and Bob could encrypt his bob-to-alice key using the GnuPG key from Alice. By regularly sending themselves new public keys, they could even establish perfect forward secrecy. I won’t implement that here, because when we get to the third part of this series, we will simply use the Freemail and Web of Trust plugin which already provide these features.

This gives us convenient, fully decentralized, spam-resistant, anonymous communication channels. Setting up a communication channel to a known person requires solving one riddle (in a real setting likely a CAPTCHA, or a password-prompt), and then the channel persists.

Note: To speed up these tests, I added another speed hack: IgnoreUSKDatehints. That turns off Date Hints, so discovering new versions will no longer be constant in the number of intermediate versions. For our messaging system that does not hurt, since we don’t have many intermediate messages we want to skip. For websites however, that could lead your visitors to see several old versions before they finally get the most current version. So be careful with this hack - just like you should with the other speed hacks.

But if we want to reach many people, we have to solve one riddle per person, which just doesn’t scale. To fix this, we can publish a list of all people we trust to be real people. Let’s do that.

Many-to-many: KSK->CAPTCHA->USK->USK which is linked in the original USK

To enable (public) many-to-many communication, we propagate the information that we believe that someone isn’t a spammer and add a blacklist to get rid of people who suddenly start to spam.

The big change with this scheme is that there is two-step authentication: Something expensive (solving a riddle) gets you seen by a few people, and if you then contribute constructively in a social context, they mark you as non-spammer and you get seen by more people.

The clever part about that scheme is that socializing is actually no cost to honest users (that’s why we use things like Sone or FMS), while it is a cost to attackers.

Let’s take Alice and Bob again, but add Carol. First Bob introduces himself to Alice, then Carol introduces herself to Alice. Thanks to propagating the riddle-information, Carol can directly write to Bob, without first solving a riddle. Scaling up that means that you only need to prove a single time that you are no spammer (or rather: not disruptive) if you want to enter a community.

To make it easier to follow, we will implement this with a bit of abstraction: People have a private key, can introduce themselves and publish lists of messages. Also they keep a public list of known people and a list of people they see as spammers who want to disrupt communication.

I got a bit carried away while implementing this, but please bear with me: It’ll work hard to make it this fun.

The finished program is available as alice_bob_carol.py. Just download and run it with python alice_bob_carol.py.

Let’s start with the minimal structure for any pyFreenet using program:

import fcp

n = fcp.node.FCPNode() # for debugging add verbosity=5

<<body>>

n.shutdown()

The body contains the definitions of a person with different actors, an update step (as simplification I use global stepwise updates) as well as the setup of the communication. Finally we need an event loop to run the system.

<<preparation>>

<<person>>

<<update>>

<<setup>>

<<event_loop>>

We start with some imports – and a bit of fun :)

import uuid
import random
try:
    import chatterbot # let's get a real conversation :)
    # https://github.com/guntherc/ChatterBot/wiki/Quick-Start
    # get with `pip install --user chatterbot`
    irc_loguri = "USK@Dtz9FjDPmOxiT54Wjt7JwMJKWaqSOS-UGw4miINEvtg,cuIx2THw7G7cVyh9PuvNiHa1e9BvNmmfTcbQ7llXh2Q,AQACAAE/irclogs/1337/"
    print "Getting the latest IRC log as base for the chatterbot"
    IRC_LOGLINES = n.get(uri=irc_loguri, realtime=True, priority=1, followRedirect=True)[1].splitlines()
    import re # what follows is an evil hack, but what the heck :)
    p = re.compile(r'<.*?>')
    q = re.compile(r'&.*?;')
    IRC_LOGLINES = [q.sub('', p.sub('', str(unicode(i.strip(), errors="ignore"))))
                    for i in IRC_LOGLINES]
    IRC_LOGLINES = [i[:-5] for i in IRC_LOGLINES # skip the time (last 5 letters)
                    if (i[:-5] and # skip empty
                        not "spam" in i # do not trigger spam-marking
                    )][7:] # skip header 
except ImportError:
    chatterbot = None

The real code begins with some helper functions – essentially data definition.

def get_usk_namespace(key, name, version=0):
    """Get a USK key with the given namespace (foldername)."""
    return "U" + key[1:] + name + "/" + str(version) + "/"

def extract_raw_from_usk(key):
    """Get an SSK key as used to identify a person from an arbitrary USK."""
    return "S" + (key[1:]+"/").split("/")[0] + "/"

def deserialize_keylist(keys_data):
    """Parse a known file to get a list of keys. Reverse: serialize_keylist."""
    return [i for i in keys_data.split("\n") if i]

def serialize_keylist(keys_list):
    """Serialize the known keys into a text file. Reverse: parse_known."""
    return "\n".join(keys_list)

Now we can define a person. The person is the primary actor. To keep everything contained, I use a class with some helper functions.

class Person(object):
    def __init__(self, myname, mymessage):
        self.name = myname
        self.message = mymessage
        self.introduced = False
        self.public_key, self.private_key = n.genkey()
        print self.name, "uses key", self.public_key
        # we need a list of versions for the different keys
        self.versions = {}
        for name in ["messages", "riddles", "known", "spammers"]:
            self.versions[name] = -1 # does not exist yet
        # and sets of answers, watched riddle-answer keys, known people and spammers.
        # We use sets for these, because we only need membership-tests and iteration.
        # The answers contain KSKs, the others the raw SSK of the person.
        # watched contains all persons whose messages we read.
        self.lists = {}
        for name in ["answers", "watched", "known", "spammers", "knowntocheck"]:
            self.lists[name] = set()
        # running requests per name, used for making all persons update asynchronously
        self.jobs = {}
        # and just for fun: get real conversations. Needs chatterbot and IRC_LOGLINES.
        # this is a bit slow to start, but fun. 
        try:
            self.chatbot = chatterbot.ChatBot(self.name)
            self.chatbot.train(IRC_LOGLINES)
        except:
            self.chatbot = None


    def public_usk(self, name, version=0):
        """Get the public usk of type name."""
        return get_usk_namespace(self.public_key, name, version)
    def private_usk(self, name, version=0):
        """Get the private usk of type name."""
        return get_usk_namespace(self.private_key, name, version)

    def put(self, key, data):
        """Insert the data asynchronously to the key. This is just a helper to
avoid typing the realtime arguments over and over again.

        :returns: a job object. To get the public key, use job.wait(60)."""
        return n.put(uri=key, data=data, async=True,
                     Global=True, persistence="forever",
                     realtime=True, priority=1,
                     IgnoreUSKDatehints="true")

    def get(self, key):
        """Retrieve the data asynchronously to the key. This is just a helper to
avoid typing the realtime arguments over and over again.

        :returns: a job object. To get the public key, use job.wait(60)."""
        return n.get(uri=key, async=True,
                     realtime=True, priority=1,
                     IgnoreUSKDatehints="true",
                     followRedirect=True)

    def introduce_to_start(self, other_public_key):
        """Introduce self to the other by solving a riddle and uploading the messages USK."""
        riddlekey = get_usk_namespace(other_public_key, "riddles", "-1") # -1 means the latest version
        try:
            self.jobs["getriddle"].append(self.get(riddlekey))
        except KeyError:
            self.jobs["getriddle"] = [self.get(riddlekey)]

    def introduce_start(self):
        """Select a person and start a job to get a riddle."""
        known = list(self.lists["known"])
        if known: # introduce to a random person to minimize
                  # the chance of collisions
            k = random.choice(known)
            self.introduce_to_start(k)

    def introduce_process(self):
        """Get and process the riddle data."""
        for job in self.jobs.get("getriddle", [])[:]:
            if job.isComplete():
                try:
                    riddle = job.wait()[1]
                except Exception as e: # try again next time
                    print self.name, "getting the riddle from", job.uri, "failed with", e
                    return
                self.jobs["getriddle"].remove(job)
                answerkey = self.solve_riddle(riddle)
                messagekey = self.public_usk("messages")
                try:
                    self.jobs["answerriddle"].append(self.put(answerkey, messagekey))
                except KeyError:
                    self.jobs["answerriddle"] = [self.put(answerkey, messagekey)]

    def introduce_finalize(self):
        """Check whether the riddle answer was inserted successfully."""
        for job in self.jobs.get("answerriddle", [])[:]:
            if job.isComplete():
                try:
                    job.wait()
                    self.jobs["answerriddle"].remove(job)
                    self.introduced = True
                except Exception as e: # try again next time
                    print self.name, "inserting the riddle-answer failed with", e
                    return

    def new_riddle(self):
        """Create and upload a new riddle."""
        answerkey = "KSK@" + str(uuid.uuid1()) + "-answered"
        self.lists["answers"].add(answerkey)
        self.versions["riddles"] += 1
        next_riddle_key = self.private_usk("riddles", self.versions["riddles"])
        self.put(next_riddle_key, answerkey)


    def solve_riddle(self, riddle):
        """Get the key for the given riddle. In this example we make it easy:
The riddle is the key. For a real system, this needs user interaction.
        """
        return riddle

    def update_info(self):
        for name in ["known", "spammers"]:
            data = serialize_keylist(self.lists[name])
            self.versions[name] += 1
            key = self.private_usk(name, version=self.versions[name])
            self.put(key, data)

    def publish(self, data):
        self.versions["messages"] += 1
        messagekey = self.private_usk("messages", version=self.versions["messages"])
        print self.name, "published a message:", data
        self.put(messagekey, data)

    def check_network_start(self):
        """start all network checks."""
        # first cancel all running jobs which will be replaced here.
        for name in ["answers", "watched", "known", "knowntocheck", "spammers"]:
            for job in self.jobs.get(name, []):
                job.cancel()
        # start jobs for checking answers, for checking all known people and for checking all messagelists for new messages.
        for name in ["answers"]:
            self.jobs[name] = [self.get(i) for i in self.lists[name]]
        for name in ["watched"]:
            self.jobs["messages"] = [self.get(get_usk_namespace(i, "messages")) for i in self.lists[name]]
        self.jobs["spammers"] = []
        for name in ["known", "knowntocheck"]:
            # find new nodes
            self.jobs[name] = [self.get(get_usk_namespace(i, "known")) for i in self.lists[name]]
            # register new nodes marked as spammers
            self.jobs["spammers"].extend([self.get(get_usk_namespace(i, "spammers")) for i in self.lists[name]])

    def process_network_results(self):
        """wait for completion of all network checks and process the results."""
        for kind, jobs in self.jobs.items():
            for job in jobs:
                if not kind in ["getriddle", "answerriddle"]:
                    try:
                        res = job.wait(60)[1]
                        self.handle(res, kind, job)
                    except:
                        continue

    def handle(self, result, kind, job):
        """Handle a successful job of type kind."""
        # travel the known nodes to find new ones
        if kind in ["known", "knowntocheck"]:
            for k in deserialize_keylist(result):
                if (not k in self.lists["spammers"] and
                    not k in self.lists["known"] and
                    not k == self.public_key):
                    self.lists["knowntocheck"].add(k)
                    self.lists["watched"].add(k)
                    print self.name, "found and started to watch", k
        # read introductions
        elif kind in ["answers"]:
            self.lists[kind].remove(job.uri) # no longer need to watch this riddle
            k = extract_raw_from_usk(result)
            if not k in self.lists["spammers"]:
                self.lists["watched"].add(k)
                print self.name, "discovered", k, "through a solved riddle"
        # remove found spammers
        elif kind in ["spammers"]:
            for k in deserialize_keylist(result):
                if not result in self.lists["known"]:
                    self.lists["watched"].remove(result)
        # check all messages for spam
        elif kind in ["messages"]:
            k = extract_raw_from_usk(job.uri)
            if not "spam" in result:
                if not k == self.public_key:
                    print self.name, "read a message:", result
                    self.chat(result) # just for fun :)
                    if not k in self.lists["known"]:
                        self.lists["known"].add(k)
                        self.update_info()
                        print self.name, "marked", k, "as known person"
            else:
                self.lists["watched"].remove(k)
                if not k in self.lists["spammers"]:
                    self.lists["spammers"].add(k)
                    self.update_info()
                    print self.name, "marked", k, "as spammer"


    def chat(self, message):
        if self.chatbot and not "spam" in self.message:
            msg = message[message.index(":")+1:-10].strip() # remove name and step
            self.message = self.name + ": " + self.chatbot.get_response(msg)

# some helper functions; the closest equivalent to structure definition
<<helper_functions>>

Note that nothing in here depends on running these from the same program. All communication between persons is done purely over Freenet. The only requirement is that there is a bootstrap key: One person known to all new users. This person could be anonymous, and even with this simple code there could be multiple bootstrap keys. In freenet we call these people “seeds”. They are the seeds from which the community grows. As soon as someone besides the seed adds a person as known, the seed is no longer needed to keep the communication going.

The spam detection implementation is pretty naive: It trusts people to mark others as spammers. In a real system, there will be disputes about what constitutes spam and the system needs to show who marks whom as spammer, so users can decide to stop trusting the spam notices from someone when they disagree. As example for a real-life system, the Web of Trust plugin uses trust ratings between -100 and 100 and calculates a score from the ratings of all trusted people to decide how much to trust people who are not rated explicitly by the user.

With this in place, we need the update system to be able to step through the simulation. We have a list of people who check keys of known other people.

We first start all checks for all people quasi-simultaneously and then check the results in serial to avoid long wait times from high latency. Freenet can check many keys simultaneously, but serial checking is slow.

people = []

def update(step):
    for p in people:
        if not p.introduced:
            p.introduce_start()
    for p in people:
        p.check_network_start()
    for p in people:
        if p.message:
            p.publish(p.name + ": " + p.message + "   (step=%s)" % step)
        p.new_riddle()
    for p in people:
        if not p.introduced:
            p.introduce_process()
    for p in people:
        p.process_network_results()
    for p in people:
        if not p.introduced:
            p.introduce_finalize()

So that’s the update tasks - not really rocket science thanks to the fleshed out Persons. Only two things remain: Setting up the scene and actually running it.

For setup: We have Alice, Bob and Carol. Lets also add Chuck who wants to prevent the others from communicating by flooding them with spam.

def gen_person(name):
    try:
        return Person(myname=name, mymessage=random.choice(IRC_LOGLINES))
    except:
        return Person(myname=name, mymessage="Hi, it's me!")

# start with alice
alice = gen_person("Alice")
people.append(alice)

# happy, friendly people
for name in ["Bob", "Carol"]:
    p = gen_person(name)
    people.append(p)

# and Chuck
p = Person(myname="Chuck", mymessage="spam")
people.append(p)

# All people know Alice (except for Alice).
for p in people:
    if p == alice:
        continue
    p.lists["known"].add(alice.public_key)
    p.lists["watched"].add(alice.public_key)

# upload the first version of the spammer and known lists
for p in people:
    p.update_info()

That’s it. The stage is set, let the trouble begin :)

We don’t need a while loop here, since we just want to know whether the system works. So the event loop is pretty simple: Just call the update function a few times.

for i in range(6):
    update(step=i)

That’s it. We have spam-resistant message-channels and community discussions. Now we could go on and implement more algorithms on this scheme, like the turn-based games specification (ever wanted to play against truly anonymous competitors?), Fritter (can you guess from its name what it is? :)), a truly privacy respecting dropbox or an anonymizing, censoriship resistant, self-hosting backend for a digital market like (the in 2023 long defunct) OpenBazaar.

But that would go far beyond the goal of this article – which is to give you, my readers, the tools to create the next big thing by harnessing the capabilities of Freenet.

These capabilities have been there for years, but hidden beneath non-existing and outdated documentation, misleading claims of being in alpha-stage even though Freenet has been used in what amounts to production for over a decade and, not to forget, the ever-recurring, ever-damning suggestion to SGTFS (second-guess the friendly source). As written in Forgotten Cypherpunk Paradise, Freenet already solved many problems which researchers only begin to tackle now, but there are reasons why it was almost forgotten. With this series I intend fix some of them and start moving Freenet documentation towards the utopian vision laid out in Teach, Don’t Tell. It’s up to you to decide whether I succeeded. If I did, it will show up as a tiny contribution to the utilities and works of art and vision you create.

Note that this is not fast (i.e. enough for blogging but not enough for chat). We can make it faster by going back to SSKs instead of USKs with their additional logic for finding the newest version in O(1), but for USK there are very cheap methods to get notified of new versions for large numbers of keys (subscribing) which are used by more advanced tools like the Web of Trust and the Sone plugin, so this would be an optimization we would have to revert later. With these methods, Sone reaches round trip times of 5-15 minutes despite using large uploads.

Also since this uses Freenet as backend, it scales up: If Alice, Bob, Carol und Chuck used different computers instead of running on my single node, their communication would actually be faster, and if they called in all their alphabet and unicode friends, the system would still run fast. We’re harvesting part of the payoff from using a fully distributed backend :)

And with that, this installment ends. You can now implement really cool stuff using Freenet. In the next article I’ll describe how to avoid doing this stuff myself by interfacing with existing plugins. Naturally I could have done that from the start, but then how could I have explained the Freenet communication primitives these plugins use? :)

If you don’t want to wait, have a look at how Infocalypse uses wot to implement github-like access with user/repo, interfaces with Freemail to realize truly anonymous pull-requests from the command line and builds on FMS to provide automated updates of a DVCS wiki over Freenet.

Happy Hacking!

PS: You might ask “What is missing?”. You might have a nagging feeling that something we do every day isn’t in there. And you’re right. It’s scalable search. Or rather: scalable, spam- and censorship-resistant search. Scalable search would be Gnutella. Spam-resistance would be Credence on the social graph (the people you communicate with). Censorship-resistant is unsolved – even Google fails there. But seeing that Facebook just overtook Google as the main source of traffic, we might not actually need fully global search. Together with the cheap and easy update notifications in Freenet (via USKs), a social recommendation and bookmark-sharing system should make scalable search over Freenet possible. And until then there’s always the decentralized YaCy search engine which has been shown to be capable of crawling Freenet. Also there are the Library and Spider plugins, but they need some love to work well. Also there are the Library and Spider plugins, but they need some love to work well.

PPS: You can download the final example as alice_bob_carol.py

Druckversion
Login to post comments

Use Node:

⚙ Babcom is trying to load the comments ⚙

This textbox will disappear when the comments have been loaded.

If the box below shows an error-page, you need to install Freenet with the Sone-Plugin or set the node-path to your freenet node and click the Reload Comments button (or return).

If you see something like Invalid key: java.net.MalformedURLException: There is no @ in that URI! (Sone/search.html), you need to setup Sone and the Web of Trust

If you had Javascript enabled, you would see comments for this page instead of the Sone page of the sites author.

Note: To make a comment which isn’t a reply visible to others here, include a link to this site somewhere in the text of your comment. It will then show up here. To ensure that I get notified of your comment, also include my Sone-ID.

Link to this site and my Sone ID: sone://6~ZDYdvAgMoUfG6M5Kwi7SQqyS-gTcyFeaNN1Pf3FvY

This spam-resistant comment-field is made with babcom.

Freenet Communication Primitives: Part 2, Service Discovery and Communication

Table of Contents

USK: The Updatable Subspace Key

KSK: Upload a file to a password

KSK queue: Share files by uploading to a password

KSK queue with CAPTCHA

Convenience: KSK queue with CAPTCHA via USK to reference a USK

Many-to-many: KSK->CAPTCHA->USK->USK which is linked in the original USK

Beliebte Inhalte

Heute:

Zuletzt angezeigt:

Draketo neu: Beiträge

Ein Würfel System

sn.1w6.org news

Sep. 12th decides the fate of the internet in the EU!