Add the Twitter and Accounts libraries to your project

Click on your project file, and then on your build target. Make sure you’re on the "build phases" tab. Under the "Link Binary With Libraries" section, click the plus symbol to the bottom left, and search for Accounts.framework and add it. Then do the same for Twitter.framework. This will link all of the necessary libraries into your project so that we can use the Twitter integration.

Requesting access to a user’s twitter account

The first thing to do when you want to let a user sign-on with Twitter is to create a long-lived (save it as an instance variable, for example) instance of ACAccountStore and request access to the Twitter accounts contained within:

1
2
3
4
5
6
7
8

ACAccountStore *store = [[ACAccountStore alloc] init]; // Long-lived
ACAccountType *twitterType = [store accountTypeWithAccountTypeIdentifier:ACAccountTypeIdentifierTwitter];
[store requestAccessToAccountsWithType:twitterType withCompletionHandler:^(BOOL granted, NSError *error) {
    if(granted) {
        // Access has been granted, now we can access the accounts
    }
    // Handle any error state here as you wish
}];

What do we do once we have access to the accounts? Well, we get a list of ‘em. If they don’t have any accounts, then we can show a dialog asking them to connect one in the iOS settings app:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

// Remember that twitterType was instantiated above
NSArray *twitterAccounts = [store accountsWithType:twitterType];

// If there are no accounts, we need to pop up an alert
if(twitterAccounts != nil && [twitterAccounts count] > 0) {
    UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"No Twitter Accounts"
                                                    message:@"There are no Twitter accounts configured. You can add or create a Twitter account in Settings."
                                                   delegate:nil
                                          cancelButtonTitle:@"OK"
                                          otherButtonTitles:nil];
    [alert show];
    [alert release];
} else {
    ACAccount *account = [twitterAccounts objectAtIndex:0];
    // Do something with their Twitter account
}

Now what?

Well, now you have an access token to read some information about the user from their Twitter stream. The vast majority of apps will just want to grab the user’s basic info to get things like a username, real name, and maybe their location. Here’s how that would look:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

NSURL *url = [NSURL URLWithString:@"http://api.twitter.com/1/account/verify_credentials.json"];
TWRequest *req = [[TWRequest alloc] initWithURL:url
                                     parameters:nil
                                  requestMethod:TWRequestMethodGET];

// Important: attach the user's Twitter ACAccount object to the request
req.account = account;

[req performRequestWithHandler:^(NSData *responseData,
                                 NSHTTPURLResponse *urlResponse,
                                 NSError *error) {

    // If there was an error making the request, display a message to the user
    if(error != nil) {
        UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"Twitter Error"
                                                        message:@"There was an error talking to Twitter. Please try again later."
                                                       delegate:nil
                                              cancelButtonTitle:@"OK"
                                              otherButtonTitles:nil];
        [alert show];
        [alert release];
        return;
    }

    // Parse the JSON response
    NSError *jsonError = nil;
    id resp = [NSJSONSerialization JSONObjectWithData:responseData
                                                      options:0
                                                        error:&jsonError];

    // If there was an error decoding the JSON, display a message to the user
    if(jsonError != nil) {
        UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"Twitter Error"
                                                        message:@"Twitter is not acting properly right now. Please try again later."
                                                       delegate:nil
                                              cancelButtonTitle:@"OK"
                                              otherButtonTitles:nil];
        [alert show];
        [alert release];
        return;
    }

    NSString *screenName = [resp objectForKey:@"screen_name"];
    NSString *fullName = [resp objectForKey:@"name"];
    NSString *location = [resp objectForKey:@"location"];

    // Make sure to perform our operation back on the main thread
    dispatch_async(dispatch_get_main_queue(), ^{
        // Do something with the fetched data
    });
}];

Most of the code here is actually error handling code. The meat of what we’re doing is simply fetching the user’s credentials from Twitter, parsing the JSON, and doing something with it. (What exactly you want to do with the username and name data is left as an excercise for the reader.)

That’s it?

Yep, that’s it. You should be able to implement Twitter sign-on for your app with the simple code I’ve shown here. The only bummer is the case when the user doesn’t have a Twitter account registered. We’re not allowed to help the user out by sending them to the Settings app. All we can do is tell users to go there and hope they can figure it out.

Any other tips or tricks you have for implementing sign-on with Twitter on iOS? Be sure to tweet me about it!

P.S. If you’re building mobile apps, my startup clutch.io can help you build and iterate faster on them. Check us out :)

What do I mean by code nesting, and why is it bad?

It’s easier to demonstrate rather than talk about it. This is what I mean by deep code nesting, with my apologies for the contrived example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

def get_cached_user(user_id=None, username=None):
    """
    Returns a cached user object either by their user_id or by their username.
    """
    user = cache.get_user_by_id(user_id)
    if not user:
        user = cache.get_user_by_username(username)
        if not user:
            user = db.get_user_by_id(user_id)
            if not user:
                user = db.get_user_by_username(username)
                if not user:
                    raise ValueError('User not found')
            cache.set_user(user, id=user.id, username=user.username)
    return user

You can see in this Python code just by looking at the indentation level that there’s lots of nesting. Before we can determine that the user was not found, we must pass through four conditionals, and each conditional is nested within the previous conditional.

I argue that this is bad code. Every added level of nesting is another piece of context that your brain has to keep track of. Each nested block is one you have to line up by eye to see what conditional it lines up with (even if your editor helps at this with visuals, it doesn’t remove the issue entirely.) And this is just a straightforward example where we just return the user at the end, let’s take a look at an example that does something more complicated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

def get_media_details(media):
    """
    Returns a dictionary of extra data about the given media object.
    """
    data = {}
    if media.is_video:
        data['kind'] = 'video'
        if media.is_youtube:
            data['url'] = 'http://youtube.com/'
        if media.is_vimeo:
            data['vimeo'] = True
            if media.vimeo_version == 2:
                data['url'] = 'http://vimeo.com/v2/'
        if 'url' in data:
            data['secure_url'] = data['url'].replace('http:', 'https:')
    elif media.is_audio:
        data['kind'] = 'audio'
    elif media.is_text:
        data['kind'] = 'text'
    if 'kind' in data:
        data['kind_verbose'] = {
            'video': 'Video Stream',
            'audio': 'Audio File',
            'text': 'Text Content',
        }[data['kind']]
    return data

It was unbelievably hard for me to even write that last example. It’s obviously contrived and such, but the point is that it’s so difficult to even understand what it’s doing. Unlike the previous example, this doesn’t simply nest and then return; it nests and then un-nests, and then nests again, and then finally returns.

How to Avoid Nesting

The best way that I’ve discovered to avoid nesting is to return early. Caching is the perfect example of this. Instead of testing for a cache failure and fetching from the database inside the conditional, check for cache success and return that early.

So this code:

1
2
3
4
5
6
7

def get_cached_user(user_id):
    user = cache.get_user_by_id(user_id)
    # The main logic all happens in this nested block
    if not user:
        user = db.get_user_by_id(user_id)
        cache.set_user_for_id(user_id, user)
    return user

Becomes this:

1
2
3
4
5
6
7
8

def get_cached_user(user_id):
    user = cache.get_user_by_id(user_id)
    if user:
        return user
    # The main logic happens outside of the nested block
    user = db.get_user_by_id(user_id)
    cache.set_user_for_id(user_id, user)
    return user

In the simple case, it doesn’t seem to improve much, but what happens if we apply this technique to our first example? It’s dramatically improved:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def get_cached_user(user_id, username):
    # First check the cache by id
    user = cache.get_user_by_id(user_id)
    if user:
        return user

    # Now check the cache by username
    user = cache.get_user_by_username(username)
    if user:
        return user

    # Both caches failed, so try hitting the db for the id
    user = db.get_user_by_id(user_id)
    if user:
        cache.set_user(user, id=user.id, username=user.username)
        return user

    # Looks like that didn't exist, try the username
    user = db.get_user_by_username(username)
    if not user:
        raise ValueError('User not found')

    # Cache our final user value for future use
    cache.set_user(user, id=user.id, username=user.username)
    return user

Not only does it make it easier to read top-to-bottom, and force us to keep track of way less context, and make our code editors do less line wrapping, but it also makes it easier to separate the blocks of code and more easily comment them.

So what other techniques can we use? It starts to depend more on the situation. Are you nesting because you’re writing a bunch of callbacks? If so, you can usually restructure your code to use named functions instead of anonymous functions. Here’s how that would might look before refactoring:

1
2
3
4
5
6
7
8
9
10
11
12

function getCachedUser(userId, callback) {
    cache.getUser(userId, function(user) {
        if(user) {
            return callback(user);
        }
        db.getUser(userId, function(user) {
            cache.setUser(userId, user, function() {
                callback(user);
            });
        });
    });
}

Note that in this example we even applied the technique of returning early in the first callback function, but as you can see there’s still a bunch of nesting going on. Now if we switch to using named functions?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

function curry(fn) {
    var slice = Array.prototype.slice;
    var args = slice.apply(arguments, [1]);
    return function () {
        return fn.apply(null, args.concat(slice.apply(arguments)));
    };
}

function final(callback, user) {
    callback(user);
}

function dbResult(callback, userId, user) {
    cache.setUser(userId, user, curry(final, callback, user));
}

function cacheResult(callback, userId, user) {
    if(user) {
        return callback(user);
    }
    db.getUser(userId, curry(dbResult, callback, userId));
}

function getCachedUser(userId, callback) {
    cache.getUser(userId, curry(cacheResult, callback, userId));
}

This is a lot better in terms of nesting. Unfortunately we had to write a helper function called curry, but that only has to be written once and can be re-used for all code written in this style. Also unfortunately I still find this kind of code difficult to follow, which is why I avoid writing much callback-style code. However, at least you can reduce the nesting. In all honesty, there are probably better ways of reducing nesting that I’m not aware of. If you can rewrite the getCachedUser function in JS in a better way, please blog it!

Another way to reduce nesting is to assign an intermediate variable. Here’s an example in Erlang of some file function that nests a case statement within another case statement.

1
2
3
4
5
6
7
8
9
10
11
12
13

do_some_file_thing(File) ->
    case file:open(File, [raw, binary, read]) of
        {ok, Fd} ->
            Start = now(),
            case process_file_data(Fd) of
                {ok, Processed} ->
                    {ok, Start, now(), Processed};
                Error ->
                    Error
            end;
        Error ->
            Error
    end.

We can assign to an intermediate "Resp" variable, and bring that second case statement out into the function’s main code block, like so:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

do_some_file_thing(File) ->
    Resp = case file:open(File, [raw, binary, read]) of
        {ok, Fd} ->
            {timestamp, now(), process_file_data(Fd)};
        Error ->
            Error
    end,
    case Resp of
        {timestamp, Start, {ok, Processed}} ->
            {ok, Start, now(), Processed};
        {timestamp, Start, Error} ->
            Error;
        Error ->
            Error
    end.

What does this all mean?

At the end of the day, this isn’t going to make or break you as a programmer. In fact, nothing I’ve mentioned even changes the code’s logic, but simply its implementation. It’s simply something to think about as you code, as you read other people’s code. Hopefully you agree with me that less nesting is an admirable goal, and you find more and more ways to achieve it.

Discuss this post on Hacker News.

The basics

All of our application code is powered by Python. Our front-end html page generation is done by Django, which we use in a surprisingly traditional way given the real-time nature of Convore as a product. Everything is assembled at once: all messages, the sidebar, and the header are all rendered on the server instead of being pulled in after-the-fact with JavaScript. All of the important data is canonically stored in PostgreSQL, including messages, topics, groups, unread counts, and user profiles. Search functionality is provided by Solr, which is interfaced into our application by way of the handy Haystack Django application.

The message lifecycle

When a new message comes into the system, first it’s parsed by a series of regular expressions designed to pull out interesting bits of information from the message. Right now all we’re looking for is username references and links (and further, whether those links point at images which should be rendered in-line.) At the end of this parsing stage, we have a structured message parse list, which is converted into JSON.

So, for example if someone posted the message:

1

@ericflo @simonw Here's how we connect/disconnect from Redis in production: http://dpaste.com/406797/

The resulting JSON parse list would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

[
    {
        "type": "username",
        "user_id": 1,
        "username": "ericflo",
        "markup": "<a href=\"/users/ericflo/\">@ericflo</a>"
    },
    {
        "type": "username",
        "user_id": 56,
        "username": "simonw",
        "markup": " <a href=\"/users/simonw/\">@simonw</a>"
    },
    {
        "type": "text",
        "markup": " Here&#39;s how we connect/disconnect from Redis in production: "
    },
    {
        "type": "url",
        "url": "http://dpaste.com/406797/",
        "markup": "<a href=\"http://dpaste.com/406797/\" target=\"_blank\">http://dpaste.com/406797/</a>"
    }
]

After this is constructed, we log all our available information about this message, and then save to the database– both the raw message as it was received, and the JSON-encoded parsed node list.

Now a task is sent to Celery (by way of Redis) notifying it that this new message has been received. This Celery task now increments the unread count for everyone who has access to the topic that the message was posted in, and then it publishes to a Redis pub/sub for the group that the message was posted to. Finally, the task scans through the message, looking for any users that were mentioned in the message, and writes entries to the database for every mention.

On the other end of that pub/sub are the many open http requests that our users have initiated, which are waiting for any new messages or information. Those all simultaneously return the new message information, at which point they reconnect again, waiting for the next message to arrive.

The real-time endpoint

Our live updates endpoint is actually a very simple and lightweight pure-WSGI Python application, hosted using Eventlet. It spawns off a coroutine for each request, and in that coroutine, it looks up all the groups that a user is a member of, and then opens a connection to Redis subscribing to all of those channels. Each of these Eventlet-hosted Python applications has the ability to host hundreds-to-thousands of open connections, and we run several instances on each of our front-end machines. It has a few more responsibilities, like marking a topic as read before it returns a response, but the most important thing is to be a bridge between the user and Redis pub/sub.

Future improvements

There are so many places where our architecture can be improved. This is our first version, and now that real users are using the system, already some of our initial assumptions are being challenged. For instance, we thought that pub/sub to a channel per group would be enough, but what that means is that everyone in a group sees the exact same events as everyone else in that group.

This means we don’t have the ability to customize each user’s experience based on their preferences–no way to put a user on ignore, filter certain messages, etc. It also means that we aren’t able to sync up a user’s experience across tabs or browsers, since we don’t really want to broadcast to everyone in the group that one user has visited a topic, thereby removing any unread messages in that topic. So going forward we’re going to have to break up that per-group pub/sub into per-user pub/sub.

Another area that could be improved is our unread counts. Right now they’re stored as rows in our PostgreSQL database, which makes it extremely easy to batch update them and do aggregate queries on them, but the number of these rows is increasing rapidly, and without some kind of sharding scheme, it will at some point become more difficult to work with such a large amount of rows. My feeling is that this will eventually need to be moved into a non-relational data store, and we’ll need to write a service layer in front of it to deal with pre-aggregating and distributing updates, but nothing is set in stone just yet.

Finally, Python may not be the best language for this real-time endpoint. Eventlet is a fantastic Python library and it allowed us to build something extremely fast that has scaled to several thousand concurrent connections without breaking a sweat on launch day, but it has its limits. There is a large body of work out there on handling a large number of open connections, using Java’s NIO framework, Erlang’s mochiweb, or node.js.

That’s all folks

We’re pretty proud of what we’ve built in a very short time, and we’re glad it has held up as well as it has on our launch day and afterwards. We’re excited about the problems we’re now being faced with, both scaling the technology, and scaling the product. I hope this article has quenched any curiosity out there about how Convore works. If there are any questions, feel free to join Convore and ask away!

(Or discuss it on Hacker News)

Asynchronous programming is too hard

Let’s first examine the straightforward implementation: an event loop. In this programming model, we have a single process with a single loop that runs continuously. Functionality is achieved by writing functions to execute small tasks quickly, and inserting those functions into that event loop. One of those functions might read some bytes from a socket, while another function might write a few bytes to a file, and yet another function might do something computational like calculating an XOR on the data that’s been buffered from that first socket.

The most important part about this event loop is that only one thing is ever happening at a time. That means that you really have to break your logic up into small chunks that can be performed incrementally. If any one of our functions blocks, it hogs the event loop and nothing else can execute during that time.

We have some really great frameworks geared towards making this event loop model easier to work with. In Python, there’s Twisted and, more recently, Tornado. In Ruby there’s EventMachine. In PERL there’s POE. What these frameworks do is twofold: provide constructs for more easily working with an event loop (e.g. Deferreds or Promises), and provide asynchronous implementations of common tasks (e.g. HTTP clients and DNS resolution).

But these frameworks stop very short of making asynchronous programming easy for two reasons. The first reason is that we really do have to completely change our coding style. Consider what it would take to render a simple blog web page with comments. Here’s some JavaScript code to demonstrate how this might work in a synchronous framework:

1
2
3
4
5
6
7
8
9

function handleBlogPostRequest(request, response, postSlug) {
    var db = new DBClient();
    var post = db.getBlogPost(postSlug);
    var comments = db.getComments(post.id);
    var html = template.render('blog/post.html',
        {'post': post, 'comments': comments});
    response.write(html);
    response.close();
}

Now here’s some JavaScript code to demonstrate how this might look in an asynchronous framework. Note several things here: We’ve specifically written this in such a way that it doesn’t become nested four levels deep. We’ve also written these callback functions inside of the handleBlogPostRequest function to take advantage of closure so as to retain access to the request and response objects, the template context, and the database client. Both the desire to avoid nesting and the closure are things that we need to think about as we write this code, that were not even considerations in the synchronous version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

function handleBlogPostRequest(request, response, postSlug) {
    var context = {};
    var db = new DBClient();
    function pageRendered(html) {
        response.write(html);
        response.close();
    }
    function gotComments(comments) {
        context['comments'] = comments;
        template.render('blog/post.html', context).addCallback(pageRendered);
    }
    function gotBlogPost(post) {
        context['post'] = post;
        db.getComments(post.id).addCallback(gotComments);
    }
    db.getBlogPost(postSlug).addCallback(gotBlogPost);
}

I’ve chosen JavaScript here to prove a point, by the way. People are very excited about node.js right now, and it’s a very cool framework, but it doesn’t hide all of the complexities involved in doing things asynchronously. It only hides some of the implementation details of the event loop.

The second reason why these frameworks fall short is because not all IO can be handled properly by a framework, and in these cases we have to resort to bad hacks. For example, MySQL does not offer an asynchronous database driver, so most of the major frameworks end up using threads to ensure that this communication happens out of band.

Given the inconvenient API, the added complexity, and the simple fact that most developers haven’t switched to using this style of programming, leads us to the conclusion that this type of framework is not a desirable final solution to the problem (even though I do concede that you can get Real Work done today using these techniques, and many people do). That being the case, what other options do we have for asynchronous programming? Coroutines and lightweight processes, which brings us to our next major problem.

Languages don’t support easier asynchronous paradigms

There are a few language constructs that, if implemented properly in modern programming languages, could pave the way for alternative methods of doing asynchronous programming that don’t have the drawbacks of the event loop. These constructs are coroutines and lightweight processes.

A coroutine is a function that can suspend and resume its execution at certain, programmatically specified, locations. This simple concept can serve to transform blocking-looking code to be non-blocking. At certain critical points in your IO library code, the low-level functions that are doing IO can choose to "cooperate". That is, it can choose to suspend execution in order for another function to resume execution and continue on.

Here’s an example (it’s Python, but fairly understandable for all I hope):

1
2
3

def download_pages():
    google = urlopen('http://www.google.com/').read()
    yahoo = urlopen('http://www.yahoo.com/').read()

Normally the way this would work is that a socket would be opened, connected to Google, an HTTP request sent, and the full response would be read, buffered, and assigned to the google variable, and then in turn the same series of steps would be taken for the yahoo variable.

Ok, now imagine that the underlying socket implementation were built using coroutines that cooperated with each other. This time, just like before, the socket would be opened and a connection would be made to Google, and then a request would be fired off. This time, however, after sending the request, the socket implementation suspends its own execution.

Having suspended its execution (but not yet having returned a value), execution continues on to the next line. The same thing happens on the Yahoo line: once its request has been fired off, the Yahoo line suspends its execution. But now there’s something else to cooperate with–there’s actually some data ready to be read on the Google socket–so it resumes execution at that point. It reads some data from the Gooogle socket, and then suspends its execution again.

It jumps back and forth between the two coroutines until one has finished. Let’s say that the Yahoo socket has finished, but the Google one has not. In this case, the Google socket just continues to read from its socket until it has completed, because there are no other coroutines to cooperate with. Once the Google socket is finally finished, the function returns with all of the buffered data.

Then the Yahoo line returns with all of its buffered data.

We’ve preserved the style of our blocking code, but we’ve used asynchronous programming to do it. Best of all, we’ve preserved our original program flow–the google variable is assigned first, and then the yahoo variable is assigned. In truth, we’ve got a smart event loop going on underneath the covers to control who gets to execute, but it’s hidden from us due to the fact that coroutines are in play.

Languages like PHP, Python, Ruby, and Perl simply don’t have built-in coroutines that are robust enough to implement this kind of behind-the-scenes transformation. So what about these lightweight processes?

Lightweight processes are what Erlang uses as its main concurrency primitive. Essentially these are processes that are mostly implemented in the Erlang VM itself. Each process has approximately 300 words of overhead and its execution is scheduled primarily by the Erlang VM, sharing no state at all amongst processes. Essentially, we don’t have to think twice about spawning a process, as it’s essentially free. The catch is that these processes can only communicate via message passing.

Implementing these lightweight processes at the VM level gets rid of the memory overhead, the context switching, and the relative sluggishness of interprocess communication provided by the operating system. Since the VM also has insight into the memory stack of each process, it can freely move or resize those processes and their stacks. That’s something that the OS simply cannot do.

With this model of lightweight processes, it’s possible to again revert back to the convenient model of using a separate process for all of our asynchronous programming needs. The question becomes this: can this notion of lightweight processes be implemented in languages other than Erlang? The answer to that is "I don’t know." To my knowledge, Erlang takes advantage of some features of the language itself (such as having no mutable data structures) in its lightweight process implementation.

Where do we go from here?

The key to moving forward is to drop the notion that developers need to learn to think about all of their code in terms of callbacks and asynchrony, as the asynchronous event loop frameworks require them to do. Over the past ten years, we can see that most developers, when faced with that decision, simply choose to ignore it. They continue to use the inferior blocking methodologies of yesteryear.

We need to look at these alternative implementations like coroutines and lightweight processes, so that we can make asynchronous programming as easy as synchronous programming. Only then will we be able to kick this synchronous addiction.

Let’s get this out of the way

I love SQL. More than even that, I love my precious ORM and being able to query for whatever information I want whenever I want it. For the vast majority of sites out there (we’re talking 99.9% of the sites out there, folks) it suits their needs very well, providing a good balance of ease of use and performance.

There’s no reason for them to switch away from SQL, and there’s no way they will. If there’s one thing I don’t like about this whole NoSQL movement, it’s the presumption that everyone who’s interested in alternative databases hates the status quo. That’s simply not true.

But we’re not talking about most sites out there, we’re not talking about the status quo, we’re talking about the few applications that need something totally different.

Tokyo Cabinet / Tokyo Tyrant

Tokyo Cabinet (and its network interface, Tokyo Tyrant) is the logical successor to Berkeley DB–a blazing fast, open-source, embeddable key-value store that does just about what you would expect from its description. It supports 3 modes of operation: hashtable mode, b-tree mode, and table mode.

(Table mode still pretty much sucks, and I’m not convinced it’s a good idea for the project since it’s added bloat and other systems like RDBMs are probably better for storing tabular data, so I’m going to skip it.)

Essentially, the API into Tokyo Cabinet is that of a gigantic associative array. You give it a key and a value, and then later, given a key, it will give you back the value you put in. Its largest assets are that it’s fast and straightforward.

If your problem is such that you have a small to medium-sized amount of data, which needs to be updated rapidly, and can be easily modeled in terms of keys and values (almost all scenarios can be rewritten in terms of keys and values, but some problems are easier to convert than others), then Tokyo Cabinet and Tokyo Tyrant are the way to go.

CouchDB

CouchDB is similar to Tokyo Cabinet in that it essentially maps keys to data, but CouchDB’s philosophy is completely different. Instead of arbitrary data, its data has structure–it’s a JSON object. Instead of only being able to query by keys, you can upload functions that index your data for you and then you can call those functions. All of this is done over a very simple REST interface.

But none of this really matters. None of these really set CouchDB apart, because you could just encode JSON data and store it in Tokyo Cabinet, you can maintain your own indexes of data fairly easily, and you can build a simple REST API in a matter of days, if not hours.

What really sets CouchDB apart from the pack is it’s innovative replication strategy. It was written in such a way that nodes which are disconnected for long periods of time can reconnect, sync with each other, and reconcile their differences in a way that no other database (since Lotus Notes?) could do.

It’s functionality that allows for interesting and new distributed types of applications and data that I think could possibly change the way we take our applications offline. I imagine that some day every computer will come with CouchDB pre-installed and it’ll be a data store that we use without even knowing that we’re using it.

However, I wouldn’t choose it for a super high scalability site with lots of data and sharding and replication and high availability and all those buzzwords, because I’m not convinced it’s the right tool for that job, but I am convinced that its replication strategy will keep it relevant for years to come.

Redis

Wow, looking at the bullet points this database seems to do just about everything, perfectly! Yeah, it’s a bit prone to hyperbole and there are some great things about it, but a lot of it is hot air. For example, it claims to support sharding but really all it does is have the client run a hash function on its key and use that to determine which server to send its value to. This is something that any database can do.

When you get down to it, Redis is a key-value store which provides a richer API than something like Tokyo Cabinet. It does more operations in memory, only periodically flushing to disk, so there’s more of a risk that you could lose data on a crash. The tradeoff is that it’s extremely fast, and it does some neat things like allow you to append a value to the end of a list of items already stored for a given key.

It also has atomic operations. This is honestly the only reason I find this project interesting, because the atomic operation support that it has means that it can be turned into a best-of-breed tally server. If you are building a server to keep real-time counts of various things, you would be remiss to overlook Redis as a very viable option.

Cassandra

It’s good to save the best for last, and that’s exactly what I’ve done as I find Cassandra to be easily the most interesting non-relational database out there today. Originally developed by Facebook, it was developed by some of the key engineers behind Amazon’s famous Dynamo database.

Cassandra can be thought of as a huge 4-or-5-level associative array, where each dimension of the array gets a free index based on the keys in that level. The real power comes from that optional 5th level in the associative array, which can turn a simple key-value architecture into an architecture where you can now deal with sorted lists, based on an index of your own specification. That 5th level is called a SuperColumn, and it’s one of the reasons that Cassandra stands out from the crowd.

Cassandra has no single points of failure, and can scale from one machine to several thousands of machines clustered in different data centers. It has no central master, so any data can be written to any of the nodes in the cluster, and can be read likewise from any other node in the cluster.

It provides knobs that can be tweaked to slide the scale between consistency and availability, depending on your particular application and problem domain. And it provides a high availability guarantee, that if one node goes down, another node will step in to replace it smoothly.

Writing about all the features of Cassandra is a whole different post, but I am convinced that its data model is rich enough to support a wide variety of applications while providing the kind of extreme scalability and high availability features that few other databases can achieve–all while maintaining a lower latency than other solutions out there.

Conclusion

There are many other non-relational databases out there: HBase and Hypertable, which are replicating Google’s BigTable despite its complexity and problems with single points of failure. MongoDB is another database that has been getting some traction, but it seems to be a jack of all trades, master of none. In short, the above databases are the ones that I find interesting right now, and I would use each of them for different use cases.

What do you all think about this whole non-relational database thing? Do you agree with my thoughts or do you think I’m full of it?

The Basics

The first component for creating an application using the flojax strategy is to break up the content that you would like to reload asynchronously into smaller fragments. As a basic example of this, let’s examine the case where there is a panel of buttons that you would like to turn into asynchronous requests instead of full page reloads.

The rendered markup for a fragment of buttons could look something like this:

1
2
3
4
5

<div class="buttons">
    <a href="/vote/up/item1/">Vote up</a>
    <a href="/vote/down/item1/">Vote down</a>
    <a href="/favorite/item1/">Add to your favorites</a>
</div>

In a templating language, the logic might look something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

<div class="buttons">
    {% if voted %}
        <a href="/vote/clear/{{ item.id }}/">Clear your vote</a>
    {% else %}
        <a href="/vote/up/{{ item.id }}/">Vote up</a>
        <a href="/vote/down/{{ item.id }}/">Vote down</a>
    {% endif %}
    {% if favorited %}
        <a href="/favorite/{{ item.id }}/">Add to your favorites</a>
    {% else %}
        <a href="/unfavorite/{{ item.id }}/">Remove from your favorites</a>
    {% endif %}
</div>

(Typically you wouldn’t use anchors to do operations that can change state on the server, so you can imagine this would be accomplished using forms. However, for demonstration and clarity purposes I’m going to leave these as links.)

Now that we have written a fragment, we can start using it in our larger templates by way of an include, which might look something like this:

1
2
3
4

...
<p>If you like this item, consider favoriting or voting on it:</p>
{% include "fragments/buttons.html" %}
...

To change this from being standard links to being asynchronously updated, we just need to annotate a small amount of data onto the relevant links in the fragment.

1
2
3
4
5
6
7
8
9
10
11
12
13

<div class="buttons">
    {% if voted %}
        <a href="/vote/clear/{{ item.id }}/" class="flojax" rel="buttons">Clear your vote</a>
    {% else %}
        <a href="/vote/up/{{ item.id }}/" class="flojax" rel="buttons">Vote up</a>
        <a href="/vote/down/{{ item.id }}/" class="flojax" rel="buttons">Vote down</a>
    {% endif %}
    {% if favorited %}
        <a href="/favorite/{{ item.id }}/" class="flojax" rel="buttons">Add to your favorites</a>
    {% else %}
        <a href="/unfavorite/{{ item.id }}/" class="flojax" rel="buttons">Remove from your favorites</a>
    {% endif %}
</div>

That’s it! At this point, all of the click events that happen on these links will be changed into POST requests, and the response from the server will be inserted into the DOM in place of this div with the class of "buttons". If you didn’t catch it, all that was done was to add the "flojax" class onto each of the links, and add a rel attribute that refers to the class of the parent node in the DOM to be replaced–in this case, "buttons".

Of course, there needs to be a server side component to this strategy, so that instead of rendering the whole page, the server just renders the fragment. Most modern Javascript frameworks add a header to the request to let the server know that the request was made asynchronously from Javascript. Here’s how the code on the server to handle the flojax-style request might look (in a kind of non-web-framework-specific Python code):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

def vote(request, direction, item_id):
    item = get_item(item_id)

    if direction == 'clear':
        clear_vote(request.user, item)
    elif direction == 'up':
        vote_up(request.user, item)
    elif direction == 'down':
        vote_down(request.user, item)

    context = {'voted': direction != 'clear', 'item': item}

    if request.is_ajax():
        return render_to_response('fragments/buttons.html', context)

    # ... the non-ajax implementation details go here

    return render_to_response('items/item_detail.html', context)

There are several advantages to writing your request handlers in this way. First, note that we were able to totally reuse the same templating logic from before–we just render out the fragment instead of including it in a larger template. Second, we have provided a graceful degradation path where users without javascript are able to interact with the site as well, albeit with a worse user experience.

That’s really all there is to writing web applications using the flojax strategy.

Implementation Details

I don’t believe that the Javascript code for this method can be easily reused, because each web application tends to have a different way of showing errors and other such things to the user. In this post, I’m going to provide a reference implementation (using jQuery) that can be used as a starting point for writing your own versions. The bulk of the work is done in a function that is called on every page load, called flojax_init.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

function flojax_clicked() {
    var link = $(this);
    var parent = link.parents('.' + link.attr('rel'));

    function successCallback(data, textStatus) {
        parent.replaceWith(data);
        flojax_init();
    }
    function errorCallback(request, textStatus, errorThrown) {
        alert('There was an error in performing the requested operation');
    }

    $.ajax({
        'url': link.attr('href'),
        'type': 'POST',
        'data': '',
        'success': successCallback,
        'error': errorCallback
    });

    return false;
}

function flojax_init() {
    $('a.flojax').live('click', flojax_clicked);
}

There’s really not a lot of code there. It POSTS to the given URL and replaces the specified parent class with the content of the response, and then re-initializes the flojax handler. The re-initialization could even be done in a smarter way, as well, by targeting only the newly inserted content. Also, you might imagine that an alert message probably wouldn’t be such a great user experience, so you could integrate error messages into some sort of Javascript messaging or growl-style system.

Extending Flojax

Often times you’ll want to do other things on the page when the asynchronous request happens. For our example, maybe there is some kind of vote counter that needs to be updated or some other messages that need to be displayed.

In these cases, I have found that using hidden input elements in the fragments can be useful for transferring that information from the server to the client. As long as the value in the hidden elements adheres to some predefined structure that your client knows about (it could even be something like JSON if you need to go that route).

If what you want can’t be done by extending the fragments in this way, then flojax isn’t the right strategy for that particular feature.

Limitations

This technique cannot solve all of the world’s problems. It can’t even solve all of the problems involved in writing an AJAX-style web application. It can, however, handle a fair amount of simple cases where all you want to do is quickly set up a way for a user’s action to replace content on a page.

Some specific examples of things that flojax can’t help with are if a user action can possibly update many items on a page, or if something needs to happen without a user clicking on a link. In these situations, you are better off coding a custom solution instead of trying to shoehorn it into the flojax workflow.

Conclusion

Writing AJAX-style web applications is usually tedious, but using the techniques that I’ve described, a large majority of the tedious work can be reduced. By using the same template code for rendering the page initially as with subsequent asynchronous requests, you ensure that code is not duplicated. By rendering HTML fragments, the client doesn’t have to go through the effort of parsing the output and converting the result into correct DOM objects. Finally, by using a few unobtrusive conventions (like the rel attribute and the flojax class), the Javascript code that a web application developer writes is able to be reused again and again.

I don’t believe that any of the details that I’m describing are new. In fact, people have been doing most of these things for years. What I think may in fact be new is the generalization of the sum of these techniques in this way. It’s still very much a work in progress, though. As I use flojax more and more, I hope to find not only places where it can be extended to cover more use cases, but also its limitations and places where it makes more sense to use another approach.

What do you think about this technique? Are you using any techniques like this for your web applications? If so, how do they differ from what I’ve described?

An Example: Music Discovery Website

This has all been very abstract, so let’s take an example: Suppose you run a music discovery website that lets you play songs online. Next to each song, you simply want to display how many times the song has been played.

One solution to that problem could be to have a play_count column on the table where the song metadata is stored. Every time someone plays the song, you could issue an UPDATE on the row and increase the play_count by one. This solution will work while your site is small, but as more and more people begin using the application, the number of writes to your database is going to kill its performance.

A much more robust and scalable solution is to append a new line to a text log file every time a song is played, and have a process run regularly to scoop up all of the log files and update those play_count fields in the database.

However, even if you have that regular process run once every hour, there’s still too great a lag time between when a user takes an action and when they see the results of that action. This is where our WSGI utility comes into play. It can serve as a realtime play counter to count the plays in between the time when the logs are analyzed and the play_count columns updated.

Song Play Counter

We can design the interface for our WSGI song play counter utility any way that we like, but I’m going to try to keep it as RESTful as I can. The interface will look like this:

• GET /song/SONGID will return the current play count of the given song
• POST /song/SONGID will increment the play count of the given song by one, and return its new value
• GET / will return a mapping of all songs registered to their respective play counts
• DELETE / will clear the whole mapping

So let’s get started. First, I always like to start with a very basic skeleton:

1
2
3

def application(environ, start_response):
    start_response('200 OK', [('content-type', 'text/plain')])
    return ('Hello world!',)

This does what you would imagine, returns Hello world! to each and every request that it receives. Not very useful, so let’s make it more interesting:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

from collections import defaultdict
counts = defaultdict(int)

def application(environ, start_response):
    global counts
    path = environ['PATH_INFO']
    method = environ['REQUEST_METHOD']
    if path.startswith('/song/'):
        song_id = path[6:]
        if method == 'GET':
            start_response('200 OK', [('content-type', 'text/plain')])
            return (str(counts[song_id]),)
        elif method == 'POST':
            counts[song_id] += 1
            start_response('200 OK', [('content-type', 'text/plain')])
            return (str(counts[song_id]),)
        else:
            start_response('405 METHOD NOT ALLOWED', [('content-type', 'text/plain')])
            return ('Method Not Allowed',)
    start_response('404 NOT FOUND', [('content-type', 'text/plain')])
    return ('Not Found',)

We’ve now added the data structure that we’re using to keep track of the counts, which in this case is a defaultdict(int). We’re also now looking at the request path and method, as well. If it’s a GET starting with /song/, we look up the count and return it, and if it’s a POST starting with /song/, we increment it by one before returning it. Also, we’re doing the proper thing if we detect a method that’s not allowed: we’re returning HTTP error code 405.

Now let’s add the final bit of functionality:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

from collections import defaultdict
counts = defaultdict(int)

def application(environ, start_response):
    # ... start of app
    if path.startswith('/song/'):
        # ... song-specific logic
    elif path == '/':
        if method == 'GET':
            res = ','.join(['%s=%s' % (k, v) for k, v in counts.iteritems()])
            start_response('200 OK', [('content-type', 'text/plain')])
            return (res,)
        elif method == 'DELETE':
            counts = defaultdict(int)
            start_response('200 OK', [('content-type', 'text/plain')])
            return ('OK',)
        else:
            start_response('405 METHOD NOT ALLOWED', [('content-type', 'text/plain')])
            return ('Method Not Allowed',)
    # ... rest of app

We’ve done basically the same thing here as we did with the previous example: we are looking at the request path and method and doing the appropriate action. There really is nothing very tricky going on here. We’re inventing our own format for the case where we return the counts for all songs, but it’s nothing that will be hard to parse.

NOTE: Generally you would want to use some sort of threading lock primitive before accessing a global dictionary like this. I will be using Spawning to run this WSGI application, with a threadpool size of 0 to use cooperative coroutines instead of standard threads, so I am able to get away without locks for this application. To install Spawning for yourself, just type:

1

sudo easy_install Spawning

Running the Utility

Let’s just take a quick look at how this utility works, from the command line:

1

$ spawn -t 0 -p 8000 counter.application

…and in another window:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ curl http://127.0.0.1:8000/song/1
0
$ curl -X POST http://127.0.0.1:8000/song/1
1
$ curl http://127.0.0.1:8000/song/1
1
$ curl -X POST http://127.0.0.1:8000/song/5
1
$ curl -X POST http://127.0.0.1:8000/song/5
2
$ curl http://127.0.0.1:8000/
1=1,5=2
$ curl -X DELETE http://127.0.0.1:8000/
OK

As you can see, it seems to be working correctly. The play counter is behaving as expected.

Writing a Client to Talk to our Utility

Now that we have our WSGI utility written to keep track of the counts on our songs, we should write a client library to communicate with this server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

import httplib

class CountClient(object):
    def __init__(self, servers=['127.0.0.1:8000']):
        self.servers = servers

    def _get_server(self, song_id):
        return self.servers[song_id % len(self.servers)]

    def _song_request(self, song_id, method):
        conn = httplib.HTTPConnection(self._get_server(song_id))
        conn.request(method, '/song/%s' % (song_id,))
        resp = conn.getresponse()
        play_count = int(resp.read())
        conn.close()
        return play_count

    def get_play_count(self, song_id):
        return self._song_request(song_id, 'GET')

    def increment_play_count(self, song_id):
        return self._song_request(song_id, 'POST')

    def get_all_play_counts(self):
        dct = {}
        for server in self.servers:
            conn = httplib.HTTPConnection(server)
            conn.request('GET', '/')
            counts = conn.getresponse().read()
            conn.close()
            if not counts:
                continue
            dct.update(dict([map(int, pair.split('=')) for pair in counts.split(',')]))
        return dct

    def reset_all_play_counts(self):
        status = True
        for server in self.servers:
            conn = httplib.HTTPConnection(server)
            conn.request('DELETE', '/')
            resp = conn.getresponse().read()
            if resp != 'OK':
                status = False
            conn.close()
        return status

What we have here is a simple class that converts Python method calls to the RESTful HTTP equivalents that we have written for our WSGI utility. The best part about this setup, though, is that it uses a hash based on the song_id to determine which server to connect to. If you only ever do per-song operations, this setup is quite literally infinitely scalable. You could have thousands of servers keeping track of song counts, none of them knowing about each other. Since the decision about which server to talk to happens on the client side, there needs to be no communication between the servers whatsoever.

However, if you start to use the get_all_play_counts and reset_all_play_counts, then eventually after many many servers are added it will start to get slower.

Let’s explore this client:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

>>> from countclient import CountClient
>>> c = CountClient()
>>> c.get_play_count(1)
0
>>> c.increment_play_count(1)
1
>>> c.increment_play_count(1)
2
>>> c.get_play_count(1)
2
>>> c.increment_play_count(5)
1
>>> c.get_all_play_counts()
{1: 2, 5: 1}
>>> c.reset_all_play_counts()
True
>>> c.get_all_play_counts()
{}

Benchmarks!

I’m not a benchmarking nut in any way, shape, or form these days. However, in Python it’s quite tough to beat pure-WSGI applications for raw speed. Using my MacBook Pro with a 2.5GHz Intel Core 2 Duo and 2 GB 667 MHz DDR2 SDRAM I got these results from ApacheBench:

e:Desktop ericflo$ ab -n 10000 http://127.0.0.1:8000/song/1
...
Concurrency Level:      1
Time taken for tests:   7.792 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1020000 bytes
HTML transferred:       10000 bytes
Requests per second:    1283.31 [#/sec] (mean)
Time per request:       0.779 [ms] (mean)
Time per request:       0.779 [ms] (mean, across all concurrent requests)
Transfer rate:          127.83 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       2
Processing:     0    1   0.8      1      43
Waiting:        0    1   0.5      0      43
Total:          1    1   0.8      1      43

Take these results with a huge grain of salt, but suffice it to say, it’s fast. It would probably be even faster using mod_wsgi instead of Spawning.

Drawing Conclusions From This Exercise

I don’t want to misconstrue my standpoint on this: frameworks definitely have their place. There’s no way you would want to write an entire user-facing application with pure WSGI unless you were using lots of middleware and stuff and at some point you’re just recreating Pylons. But when you’re writing a HTTP utility like we did here, then I think that pure-WSGI is the way to go.

I’d like to touch on one more nice side effect of using pure-WSGI: You can run it in any application server that supports WSGI. That means Google App Engine, Apache, Spawning, CherryPy, and many other containers. It can easily be served by pure python so even on very restrictive shared hosting it’s possible to run your utility.

What do you think of pure-WSGI utilities? Are you using them in your app? I’d love to hear about it–leave me a comment and tell me your thoughts on this subject.

2008 in Review

What a fantastic year. What a crazy year. I don’t think there’s ever been a year in my life so filled with change. For starters, I…

• Learned Erlang
• Joined Twitter and FriendFeed
• Drove 3 hours to caucus in Iowa (what a strange ritual that is!)
• Met some really cool people at PyCon 2008
• Became involved in the Pinax project
• Wrote and released my first 7 open source Django applications
• Guest hosted a podcast
• Learned how to use Git
• Graduated from college with bachelor’s degree in Computer Science
• Accepted a job offer with Mochi Media
• Moved to San Francisco
• Released a series of 14 screencasts building a project from the ground up
• Wrote one blog post every day in the month of November
• Quit drinking soda
• Participated in the most notable election of my lifetime
• Started a Django San Francisco local user group

…but those are just the highlights. It’s been really interesting transitioning from college student to the real world. Somehow I thought that not having homework would mean that I would have more free time to do other things, but it turns out that you actually have less time if you have a job to go to every day.

But I don’t think that being a student ever stopped. I’ve learned so much from my coworkers: different ways of thinking about problems, theoretical problems, real world challenges, prioritizing work, RESTful principles of the web, and a whole lot about databases. This is what I like about this industry. However much you know, there’s more to learn, and people are generally willing to share their expertise with you.

I’ve learned a lot about what it means to be part of a community; part of a company. I’ve learned a lot about life, this year.

But the learning’s not over.

Goals for 2009

While I’m weary of posting my goals for all the world to see, I think it’s important to codify them and make them public. That way, maybe it’ll compel me to actually follow through on these goals. I’ve tried to keep them realistic and with a few exceptions, specific enough to be testable. Without further ado, here are my goals for 2009 (in no particular order):

• Learn a concatenative/stack-based language
• Meet new people, especially those that I wouldn’t normally meet
• Write a virtual machine
• Double both my blog and Twitter followership
• Be more consistent in my contributions to the Pinax project, instead of helping in fits and spurts like I currently am
• Post at least twice a month to this blog
• Release some non-open source software, and maybe even charge for it
• Work out an average of 2 times a week or more over the whole year
• Learn Processing
• Learn about investments, and invest 10% of my earnings each month
• Give a talk at a conference

I think that all of these goals are achievable. In fact, I’d like to do more than just what I listed here, but these are the ones I’m willing to commit to. Frankly, I can’t wait to see how this year shapes up. If it turns out to be anything like last year, I’m in for quite a ride.

CouchDB is schema-free

One of the most annoying parts of dealing with a traditional SQL database is that you invariably need to change your schemata. This can be done usually with some ALTER TABLE statements, but other times it requires scripts and careful use of transactions, etc. In CouchDB, the solution is to just start using your new schema. No migration needed. If it’s a significant change, then you might need to change your views slightly, but nothing as annoying as what would be needed with SQL.

The other advantage of having no schema is that some types of data just aren’t well suited to having a strict schema enforced upon them. My CouchDB-based lifestreaming application is a perfect example of the inherent flexibility of CouchDB’s schemaless design is that all kinds of disparate information can be stored alongside each other and sorted and aggregated. There’s also no reason that you need to use its schema-free nature this way. You could, for example, manually enforce a schema for certain databases, if needed.

CouchDB is RESTFUL HTTP

When is the last time you tried to install MySQL or PostgreSQL drivers for your web development platform of choice? If you’re using apt-get it’s not so bad, but for just about every other platform, it’s a total pain to get these drivers up and running. With CouchDB, there’s no need. It speaks HTTP. Want to create a new database? Send an HTTP PUT request. Want to retrieve a document from the database? Send an HTTP GET. Want to delete a database? Send an HTTP DELETE. As you can see, the API is quite straightforward and if a client library doesn’t already exist for your language of choice (hint: it does), then it will take you only a few minutes to write one.

But the best part about this is that we already have so many amazing and well-tested tools to deal with HTTP. For example, let’s say you want to store one database on one server and another database on another server? It’s as simple as setting up nginx or perlbal or varnish as a reverse proxy and having each URL go to a different machine. The same thing goes for transparent caching, etc. Oh, and also, web browsers know how to speak HTTP, too. You could easily write whole web apps served only from CouchDB.

Map/Reduce

Map/Reduce will kill every traditional data warehousing vendor in the market. Those who adapt to it as a design/deployment pattern will survive, the rest won’t.

Sounds like someone from Google must have said this, or some Hadoop evangelist, or maybe someone who works on CouchDB. In fact, this comes from Brian Aker, a MySQL hacker who was Director of Architecture at MySQL AB and is now developing the open source fork of MySQL named Drizzle (also a very exciting project in its own right). He’s right, too. Google was on to something in a big way when they unveiled their whitepaper on Map/Reduce. It’s not the be-all end-all for processing and generating large data sets, but it certainly is a proven technology for that task.

Brian talks about massively multi-core machines which seem the inevitability these days, and we will need to start writing logic that is massively parallelizable to take advantage of these masses of CPUs. Map/Reduce is one way to force ourselves to write logic that can be parallelized. It is a good choice for any new database system to adopt for this reason, and that’s why it’s great to see that CouchDB has adopted it. It’s just one more reason why CouchDB rocks.

So much more

I could talk about how it can handle 2,500 concurrent requests in 10mb of resident memory usage. I could talk about its pluggable view server backends, so that instead of writing views in JavaScript you can write them in Python or any other language (given the correct bindings). I could talk about CouchDBX, which makes installing it on the Mac, quite literally, one click. I could even talk about how it’s written in Erlang, with an eye towards scalability. Or maybe about how its database store is append-only.

I could talk about any of those things, and more. It just comes down to this: CouchDB rocks. But don’t take my word for it–try it out for yourself!