Apple has included integration with Twitter in iOS5, which can be really handy to allow your users to easily tweet or log in. The only problem is that, as far as documentation of this feature is concerned, you're largely on your own. This blog post is an attempt to correct that, at least in the case of sign-on.

ACAccountStore *store = [[ACAccountStore alloc] init]; // Long-lived
ACAccountType *twitterType = [store accountTypeWithAccountTypeIdentifier:ACAccountTypeIdentifierTwitter];
[store requestAccessToAccountsWithType:twitterType withCompletionHandler:^(BOOL granted, NSError *error) {
    if(granted) {
        // Access has been granted, now we can access the accounts
    }
    // Handle any error state here as you wish
}];

// Remember that twitterType was instantiated above
NSArray *twitterAccounts = [store accountsWithType:twitterType];

// If there are no accounts, we need to pop up an alert
if(twitterAccounts != nil && [twitterAccounts count] == 0) {
    UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"No Twitter Accounts"
                                                    message:@"There are no Twitter accounts configured. You can add or create a Twitter account in Settings."
                                                   delegate:nil
                                          cancelButtonTitle:@"OK"
                                          otherButtonTitles:nil];
    [alert show];
    [alert release];
} else {
    ACAccount *account = [twitterAccounts objectAtIndex:0];
    // Do something with their Twitter account
}

NSURL *url = [NSURL URLWithString:@"http://api.twitter.com/1/account/verify_credentials.json"];
TWRequest *req = [[TWRequest alloc] initWithURL:url
                                     parameters:nil
                                  requestMethod:TWRequestMethodGET];

// Important: attach the user's Twitter ACAccount object to the request
req.account = account;

[req performRequestWithHandler:^(NSData *responseData,
                                 NSHTTPURLResponse *urlResponse,
                                 NSError *error) {

    // If there was an error making the request, display a message to the user
    if(error != nil) {
        UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"Twitter Error"
                                                        message:@"There was an error talking to Twitter. Please try again later."
                                                       delegate:nil
                                              cancelButtonTitle:@"OK"
                                              otherButtonTitles:nil];
        [alert show];
        [alert release];
        return;
    }

    // Parse the JSON response
    NSError *jsonError = nil;
    id resp = [NSJSONSerialization JSONObjectWithData:responseData
                                                      options:0
                                                        error:&jsonError];

    // If there was an error decoding the JSON, display a message to the user
    if(jsonError != nil) {
        UIAlertView *alert = [[UIAlertView alloc] initWithTitle:@"Twitter Error"
                                                        message:@"Twitter is not acting properly right now. Please try again later."
                                                       delegate:nil
                                              cancelButtonTitle:@"OK"
                                              otherButtonTitles:nil];
        [alert show];
        [alert release];
        return;
    }

    NSString *screenName = [resp objectForKey:@"screen_name"];
    NSString *fullName = [resp objectForKey:@"name"];
    NSString *location = [resp objectForKey:@"location"];

    // Make sure to perform our operation back on the main thread
    dispatch_async(dispatch_get_main_queue(), ^{
        // Do something with the fetched data
    });
}];

"This guy's code sucks!" It's something we've all said or thought when we run into code we don't like. Sometimes it's because it's buggy, sometimes it's because it conforms to a style we don't like, and sometimes it's because it just feels wrong. Recently I found myself thinking this, and automatically jumping to the conclusion that the developer who wrote it was a novice. The code had a distinct property that I dislike: lots of nesting. But the more I think about it, the more I realized that it's not really something I've heard discussed much.

So let's talk about it. I'm going to first talk about what I mean by nesting, why I think it's a bad quality, and then I'm going to go over some tricks I've learned over the years to reduce it.

def get_cached_user(user_id=None, username=None):
    """
    Returns a cached user object either by their user_id or by their username.
    """
    user = cache.get_user_by_id(user_id)
    if not user:
        user = cache.get_user_by_username(username)
        if not user:
            user = db.get_user_by_id(user_id)
            if not user:
                user = db.get_user_by_username(username)
                if not user:
                    raise ValueError('User not found')
            cache.set_user(user, id=user.id, username=user.username)
    return user

def get_media_details(media):
    """
    Returns a dictionary of extra data about the given media object.
    """
    data = {}
    if media.is_video:
        data['kind'] = 'video'
        if media.is_youtube:
            data['url'] = 'http://youtube.com/'
        if media.is_vimeo:
            data['vimeo'] = True
            if media.vimeo_version == 2:
                data['url'] = 'http://vimeo.com/v2/'
        if 'url' in data:
            data['secure_url'] = data['url'].replace('http:', 'https:')
    elif media.is_audio:
        data['kind'] = 'audio'
    elif media.is_text:
        data['kind'] = 'text'
    if 'kind' in data:
        data['kind_verbose'] = {
            'video': 'Video Stream',
            'audio': 'Audio File',
            'text': 'Text Content',
        }[data['kind']]
    return data

def get_cached_user(user_id):
    user = cache.get_user_by_id(user_id)
    # The main logic all happens in this nested block
    if not user:
        user = db.get_user_by_id(user_id)
        cache.set_user_for_id(user_id, user)
    return user

def get_cached_user(user_id):
    user = cache.get_user_by_id(user_id)
    if user:
        return user
    # The main logic happens outside of the nested block
    user = db.get_user_by_id(user_id)
    cache.set_user_for_id(user_id, user)
    return user

def get_cached_user(user_id, username):
    # First check the cache by id
    user = cache.get_user_by_id(user_id)
    if user:
        return user

    # Now check the cache by username
    user = cache.get_user_by_username(username)
    if user:
        return user

    # Both caches failed, so try hitting the db for the id
    user = db.get_user_by_id(user_id)
    if user:
        cache.set_user(user, id=user.id, username=user.username)
        return user

    # Looks like that didn't exist, try the username
    user = db.get_user_by_username(username)
    if not user:
        raise ValueError('User not found')

    # Cache our final user value for future use
    cache.set_user(user, id=user.id, username=user.username)
    return user

function getCachedUser(userId, callback) {
    cache.getUser(userId, function(user) {
        if(user) {
            return callback(user);
        }
        db.getUser(userId, function(user) {
            cache.setUser(userId, user, function() {
                callback(user);
            });
        });
    });
}

function curry(fn) {
    var slice = Array.prototype.slice;
    var args = slice.apply(arguments, [1]);
    return function () {
        return fn.apply(null, args.concat(slice.apply(arguments)));
    };
}

function final(callback, user) {
    callback(user);
}

function dbResult(callback, userId, user) {
    cache.setUser(userId, user, curry(final, callback, user));
}

function cacheResult(callback, userId, user) {
    if(user) {
        return callback(user);
    }
    db.getUser(userId, curry(dbResult, callback, userId));
}

function getCachedUser(userId, callback) {
    cache.getUser(userId, curry(cacheResult, callback, userId));
}

do_some_file_thing(File) ->
    case file:open(File, [raw, binary, read]) of
        {ok, Fd} ->
            Start = now(),
            case process_file_data(Fd) of
                {ok, Processed} ->
                    {ok, Start, now(), Processed};
                Error ->
                    Error
            end;
        Error ->
            Error
    end.

do_some_file_thing(File) ->
    Resp = case file:open(File, [raw, binary, read]) of
        {ok, Fd} ->
            {timestamp, now(), process_file_data(Fd)};
        Error ->
            Error
    end,
    case Resp of
        {timestamp, Start, {ok, Processed}} ->
            {ok, Start, now(), Processed};
        {timestamp, Start, Error} ->
            Error;
        Error ->
            Error
    end.

We launched Convore last week, and the first question developers tend to ask when they find Convore is "what technology powers this site?" It is asked so often, in fact, that we have started to copy and paste the same short response again and again. That response was good enough to satisfy people who simply wanted to know if we were Rails or Django, or whether we were using node.js for the real-time stuff, but this article will expand upon that-- not only giving more details for the curious, but also giving us a link to point people at when they ask the question in the future. I always wish other people were totally open about their architectures, so that I can learn from their good choices and their bad, so I'd like to be as open as possible about ours. Let's dive in!

@ericflo @simonw Here's how we connect/disconnect from Redis in production: http://dpaste.com/406797/

[
    {
        "type": "username",
        "user_id": 1,
        "username": "ericflo",
        "markup": "<a href=\"/users/ericflo/\">@ericflo</a>"
    },
    {
        "type": "username",
        "user_id": 56,
        "username": "simonw",
        "markup": " <a href=\"/users/simonw/\">@simonw</a>"
    },
    {
        "type": "text",
        "markup": " Here&#39;s how we connect/disconnect from Redis in production: "
    },
    {
        "type": "url",
        "url": "http://dpaste.com/406797/",
        "markup": "<a href=\"http://dpaste.com/406797/\" target=\"_blank\">http://dpaste.com/406797/</a>"
    }
]

In the moments leading up to my DjangoCon keynote this year (called Why Django Sucks, and How We Can Fix It), I pictured the room an hour in the future and imagined what things would be like. I envisioned all kinds of scenarios: one where people were literally booing, or another where I needed to sneak back to my hotel room to avoid embarassment. I imagined all of these scenarios because I was about to level some harsh criticism at a technology that everyone in the room was enthusiastic about.

And then I went on stage and gave the talk.

When it was finished, I braced myself for one of these scenarios to unfold. Instead, what happened in the following moments, the following days, and indeed the following month since then has been what I consider to be the most gracious and useful response to criticism that I've ever seen. And I think it deserves to be highlighted, because I believe that reacting well to criticism is vital for any successful community.

So what happened in the moments directly following the talk? Russell Keith-Magee, arguably the most active core developer at the time, made a special point to march up to the front of the stage and publicly shake my hand. This gesture not only legitimized some of the things that I said (some of which were quite extreme), but it also set the tone for the discourse around my talk; civility.

Afterwards, privately, I apologized to several of the Django core developers. These are people I respect and admire, and I wanted to make sure that they knew that, at least on my end, it wasn't personal. I was expecting forgiveness, but instead I received thanks. Thanking me for criticising their project? Not what I was expecting.

One of the things that I suggested was to make Alex Gaynor a core developer. A few people at the conference made comments wondering how many days it would take before that would happen. Days went by, and the conference ended without this happening. A week went by. A few more. But I knew better, because the Django core committers don't do anything brashly. They took their time, let all dust and emotions settle down, and listened to the discussion.

And then Jacob Kaplan-Moss issued an official announcement. I particularly agreed with one of the top comments on hackernews:

"I don't know enough about the specifics of the case to
assess the new policy on the merits, but I will say that
as a piece of writing this is a model of how to change
policy gracefully: clear, forthright, and non-defensive."

Clear. Forthright. Non-defensive.

Some people saw this as an opening to change even more about Django, and even when some of the discussion was phrased rudely or accusatory, the Django team's responses were thoughtful and calm. Now, over a week later (again demonstrating that things aren't done brashly), we see that a half-dozen new people have been added to the core team. Yes, that includes Alex Gaynor. So not only was the new policy clear, forthright, and non-defensive, but there was concrete follow-through so that everyone can see that it wasn't just words.

And even in that short time since the new committers were added, Django's code is already reaping the benefits--we've seen a flurry of bugs fixed and tickets closed. Django's future has never looked so promising!

Why do I think this is so important? Because a community which welcomes constructive criticism is one in which people feel they can make a difference. And it's because they can make a difference. It's a community that's never satisfied with the status quo. It's a community that can grow and change and adapt, when the world around it changes. It's a community that we can be proud of.

NB: I was not the only one to criticize Django at DjangoCon. I was merely one voice. This is just written from my perspective.

Node.js is currently at the center of a huge cyclone of hype. At this point it's clear that Node is going to be a major player in the next few years of web development. It's no wonder, either! It represents a fresh start, one with no legacy synchronous baggage in our increasingly asynchronous, increasingly real-time web. And it's accessible to anyone who's written JavaScript (read: all web developers.)

Oh, and those benchmarks! Compared trivially to the currently popular web technologies, it seems an obvious leap forward.

Yet one of Node's advantages that you'll hear repeated again and again by its proponents is that you can now code in One Language, and you won't have to deal with the cognitive load of context switching between different languages. Especially as ORMs and NoSQL continue to rise in popularity, there's no need to even deal with SQL. At the end of the day you're writing JavaScript, HTML, and CSS, and that's it.

Seeing this happen with all of these pieces falling into place--a fresh start with a unified language--I started to get excited. This would change the way we thought about our web code. The frontend is the backend is the query language is the storage layer is JavaScript! It's a revolution.

And then I saw what people did with this opportunity. They effectively ported Sinatra/Django/Rails to JavaScript--and did it in such a way that it would only run on the server, with a specific feature set of JavaScript that only Node can reasonably understand.

Not exactly the revolution I was hoping for.

Instead of coding in one language, we're actually coding in two. One is the subset JavaScript that can be run in all browsers, and another is the set of JavaScript that can be run by Node. Knowing the difference between the two languages and context switching between them is simply a required skill.

You know what would be awesome? If we wrote our libraries so that they could run either on the server or on the client, and they did so in a transparent way. Maybe it would help to give a concrete example of how this could be awesome. Let's talk about HTML templating.

Imagine a framework where the first page-load was always rendered server-side, meaning the client gets a single fully-rendered page. Then for desktop browsers, browsing around the site just made calls to API endpoints returning JSON or XML, and the client rendered the templates for the changed portions of the page. For mobile browsers with less power or for search engines, the rendering would always be done on the server. Imagine that the templating library could record some key metrics to determine how long things were taking to render, and dynamically switch between rendering on the server and client based on server load or client speed.

Imagine a case where a back-end service fails temporarily. In this case the rendering of that particular component could be deferred, the browser could be told to poll a resource. When the back-end service is recovered, it could send the data for the client to render on its own.

How awesome would that be?

It's not just HTML templating, either. This same principle could be applied to any number of things: URL routing, form validation, hell even most application logic could be done using this style.

But it's going to take discipline. Instead of reaching for those fancy V8 features, code will need to be written in a strict subset of the JavaScript that's available. Maybe Node could detect incompatible code and throw warnings, that would be cool.

I just really hope that someday we stop re-inventing the same exact wheel, and instead build something substantially different and better.

When I say I'm developing an "open" API, what does that mean to you?

To Facebook, it means something where you can get at some of your own data, as long as you abide by their restrictions on what you can do with it.

To Twitter, it means something where you get back what you put in.

To Flickr, it means something where you can get every single piece of data that you've ever put in the system, enriched by any value that Flickr can provide.

To Google, it means something where you can get back what you put in, you can host your own version of the API on your own server, and your server has exactly the same status as any Google server has, and that Google's servers and your server will interact and work as a system.

We need new words. Words that have more specific meaning--which encompass the broad idea of "open", yet remain accessible and recognizable by everyone. Federated is a good word, but it's possible to have a closed, federated system.

What words should we use?

Asynchronous programming is superior both in memory usage and in overall throughput when compared to synchronous programming . We've known this fact for years. If we look at Django or Ruby on Rails, arguably the two most promising new web application frameworks to emerge in the past few years, both of them are written in such a way that synchronous programming is assumed. Why is it that even in 2010 we're still writing programs that rely on synchronous programming ?

The reason that we're stuck on synchronous programming is twofold. Firstly, the programming model required for straightforward asynchronous implementations is inconvenient. Secondly, popular and/or mainstream languages lack the built-in language constructs that are needed to implement a less-straightforward approach to asynchronous programming.

function handleBlogPostRequest(request, response, postSlug) {
    var db = new DBClient();
    var post = db.getBlogPost(postSlug);
    var comments = db.getComments(post.id);
    var html = template.render('blog/post.html',
        {'post': post, 'comments': comments});
    response.write(html);
    response.close();
}

function handleBlogPostRequest(request, response, postSlug) {
    var context = {};
    var db = new DBClient();
    function pageRendered(html) {
        response.write(html);
        response.close();
    }
    function gotComments(comments) {
        context['comments'] = comments;
        template.render('blog/post.html', context).addCallback(pageRendered);
    }
    function gotBlogPost(post) {
        context['post'] = post;
        db.getComments(post.id).addCallback(gotComments);
    }
    db.getBlogPost(postSlug).addCallback(gotBlogPost);
}

def download_pages():
    google = urlopen('http://www.google.com/').read()
    yahoo = urlopen('http://www.yahoo.com/').read()

I recently had a conversation with a few friends that I know through the Python community. These are people who I respect a great deal, and look to for advice and insight when it comes to web development. During the course of this conversation, the subject of Facebook came up. What I wasn't expecting at all was that a large number of them said that they have no interest in Facebook, and quite a few of them proudly didn't even have accounts.

To put it bluntly, I believe that ignorance of Facebook is a major handicap today, both for developers and for entrepreneurs, and people who are not paying attention to it are burying their head in the sand.

Facebook has gotten to a point where it's the destination where the most time is spent online. It gets an estimated 260 billion (that's a B on that number) pageviews per month. In a single week, new properties are able to garner 8.6 million new active users. Sites which implement Facebook Connect typically see a 1.3x-2x boost in registrations. Companies building on Facebook's platform are being sold for 400 million dollars and are making 100 million dollars yearly. There are over 300 million active registered users. In other words, if you talk to a person that spends any time on the internet at all, they likely have a Facebook account. These numbers, by the way, are trending up and to the right--.

Take the word Facebook out of the above paragraph, and replace it with icanhazcheeseburger. Replace it with 4chan, or somethingawful, or barbie, or anything. No matter what you replace as the company name, it doesn't change the fact that those numbers are compelling enough to be irresponsible to ignore.

Let me be perfectly clear: I don't particularly enjoy using Facebook. I find its UI cluttered, its privacy controls confusing, and its content fairly trivial. From the development side, Facebook's APIs are clunky at best. I'm definitely not advocating that you should log in every day and love it. I don't. The mass populous, however, does. I'm simply advocating that you should at least have an account, log in every once in a while, and keep tabs on the announcements that they make for developers.

As people spend more and more time consuming and producing content inside of the Facebook ecosystem, it's going to be those who change and embrace it who succeed, and those who fail to adapt that stand to lose. Note that I said producing content "inside of the Facebook ecosystem", and not "inside Facebook", as Facebook has been making very strong pushes to extend the reach of its platform well outside of its destination site. You can augment your site to have people pushing content into Facebook from your own destination sites.

Facebook is willing to give you access to a person's information, to their entire social graph, and to allow your user to become an advocate of your site, all while making registration on your site as simple as one click. Sure, you may evaluate all of that as an option and decide that it's not important enough to implement, but surely it's an important enough option to warrant your cognizance.

Do I think that Facebook will be relevant in 10 years? Probably. I'm not willing to bet much on it. One thing you can be absolutely sure about though, is that if another service with this much power comes along, I'll have an account and be acutely aware of its developer resources.

Over the past few years, relational databases have fallen out of favor for a number of influential people in our industry. I'd like to weigh in on that, but before doing so, I'd like to give my executive summary of the events leading up to this movement:

In the late nineties and early thousands, websites were mostly read-only--a publisher would create some content and users would consume that content. The data access patterns for these types of applications became very well-understood, and as a result many tools were created and much research and development was done to further develop these technologies.

As the web has grown more social, however, more and more it's the people themselves who have become the publishers. And with that fundamental shift away from read-heavy architectures to read/write and write-heavy architectures, a lot of the way that we think about storing and retrieving data needed to change.

Most people have done this by relying less on the features provided by traditional relational databases and engineering more database logic in their application code. Essentially, they stop using relational databases the way they were intended to be used, and they instead use them as dumb data stores.

Other people have engineered new database systems from the ground up, each with a different set of tradeoffs and differences from their relational database brethren. It's these new databases that have some in our industry excited, and it's these databases that I'm going to focus on primarily in this post.

(By the way, there's a whole lot more theory behind the movement away from SQL. Primarily of interest is the CAP theorem and the Dynamo paper. Both of these illustrate the necessary tradeoffs of between different approaches to designing databases.)

Writing AJAX-style web applications can be very tedious. If you're using XML as your transport layer, you have to parse the XML before you can work with it. It's a bit easier if you're using JSON, but once you have parsed the data, the data still needs to be turned into HTML markup that matches the current markup on the page. Finally, the newly created markup needs to be inserted into the correct place in the DOM, and any event handlers need to be attached to the appropriate newly-inserted markup.

So there's the parsing, the markup assembly, the DOM insertion, and finally the event handler attachment. Most of the time, people tend to write custom code for each element that needs asynchronous updating. There are several drawbacks with this scenario, but the most frustrating part is probably that the presentation logic is implemented twice--once in a templating language on the server which is designed specifically for outputting markup, and again on the client with inline Javascript. This leads to problems both in the agility and in the maintainability of this type of application.

With flojax, this can all be accomplished with one generalized implementation. The same server-side logic that generates the data for the first synchronous request can be used to respond to subsequent asynchronous requests, and unobtrusive attributes specify what to do for the rest.

<div class="buttons">
    <a href="/vote/up/item1/">Vote up</a>
    <a href="/vote/down/item1/">Vote down</a>
    <a href="/favorite/item1/">Add to your favorites</a>
</div>

<div class="buttons">
    {% if voted %}
        <a href="/vote/clear/{{ item.id }}/">Clear your vote</a>
    {% else %}
        <a href="/vote/up/{{ item.id }}/">Vote up</a>
        <a href="/vote/down/{{ item.id }}/">Vote down</a>
    {% endif %}
    {% if favorited %}
        <a href="/favorite/{{ item.id }}/">Add to your favorites</a>
    {% else %}
        <a href="/unfavorite/{{ item.id }}/">Remove from your favorites</a>
    {% endif %}
</div>

...
<p>If you like this item, consider favoriting or voting on it:</p>
{% include "fragments/buttons.html" %}
...

<div class="buttons">
    {% if voted %}
        <a href="/vote/clear/{{ item.id }}/" class="flojax" rel="buttons">Clear your vote</a>
    {% else %}
        <a href="/vote/up/{{ item.id }}/" class="flojax" rel="buttons">Vote up</a>
        <a href="/vote/down/{{ item.id }}/" class="flojax" rel="buttons">Vote down</a>
    {% endif %}
    {% if favorited %}
        <a href="/favorite/{{ item.id }}/" class="flojax" rel="buttons">Add to your favorites</a>
    {% else %}
        <a href="/unfavorite/{{ item.id }}/" class="flojax" rel="buttons">Remove from your favorites</a>
    {% endif %}
</div>

def vote(request, direction, item_id):
    item = get_item(item_id)

    if direction == 'clear':
        clear_vote(request.user, item)
    elif direction == 'up':
        vote_up(request.user, item)
    elif direction == 'down':
        vote_down(request.user, item)

    context = {'voted': direction != 'clear', 'item': item}

    if request.is_ajax():
        return render_to_response('fragments/buttons.html', context)

    # ... the non-ajax implementation details go here

    return render_to_response('items/item_detail.html', context)

function flojax_clicked() {
    var link = $(this);
    var parent = link.parents('.' + link.attr('rel'));

    function successCallback(data, textStatus) {
        parent.replaceWith(data);
        flojax_init();
    }
    function errorCallback(request, textStatus, errorThrown) {
        alert('There was an error in performing the requested operation');
    }

    $.ajax({
        'url': link.attr('href'),
        'type': 'POST',
        'data': '',
        'success': successCallback,
        'error': errorCallback
    });

    return false;
}

function flojax_init() {
    $('a.flojax').live('click', flojax_clicked);
}

Recently I've been spending some quality time trying to decrease page load times and decrease the number of database accesses on a site I'm working on. As you would probably suspect, that means dealing with caching. One common thing that I need to do, however, is invalidate a large group of cache keys when some action takes place. I've devised a pattern for doing this, and while I'm sure it's not novel, I haven't seen any recent write-ups of this technique. The base idea is that we're going to add another thin cache layer, and use the value from that first layer in the key to the second layer.

First, let me give a concrete example of the problem that I'm trying to solve. I'm going to use Django/Python from here on in, but you could substitute anything else, as this pattern should work across other frameworks and even other languages.

import datetime
from django.db import models

class Favorite(models.Model):
    user = models.ForeignKey(User)
    item = models.ForeignKey(Item)
    date_added = models.DateTimeField(default=datetime.datetime.now)

    def __unicode__(self):
        return u'%s has favorited %s' % (self.user, self.item)

Given this model, now let's say that we have a function that gets the Favorite instances for a given user, which might look like this:

def get_favorites(user, start=None, end=None):
    faves = Favorite.objects.filter(user=user)
    return list(faves[start:end])

There's not much here yet--we're simply filtering to only include the Favorite instances for the given user, slicing it based on the given start and end numbers, and forcing evaluation before returning a list. Now let's start thinking about how we will cache this. We'll start by just implementing a naive cache strategy, which in this case simply means that the cache is never invalidated:

from django.core.cache import cache

def get_favorites(user, start=None, end=None):
    key = 'get_favorites-%s-%s-%s' % (user.id, start, end)
    faves = cache.get(key)
    if faves is not None:
        return faves
    faves = Favorite.objects.filter(user=user)[start:end]
    cache.set(key, list(faves), 86400 * 7)
    return faves

Now we come to the hard part: how do we invalidate those cache keys? It's especially tricky because we don't know exactly what keys have been created. What combinations of start/end have been given? We could invalidate all combinations of start/end up to some number, but that's horribly inefficient and wasteful. So what do we do? My solution is to introduce another layer. Let me explain with code:

import uuid
from django.core.cache import cache

def favorite_list_hash(user):
    key = 'favorite-list-hash-%s' % (user.id,)
    cached_key_hash = cache.get(key)
    if cached_key_hash:
        key_hash = cached_key_hash
    else:
        key_hash = str(uuid.uuid4())
        cache.set(key, key_hash, 86400 * 7)
    return (key_hash, not cached_key_hash)

Essentially what this gives us is a temporary unique identifier for each user, that's either stored in cache or generated and stuffed into the cache. How does this help? We can use this identifier in the keys to the get_favorites function:

from django.core.cache import cache

def get_favorites(user, start=None, end=None):
    key_hash, created = favorite_list_hash(user)
    key = 'get_favorites-%s-%s-%s-%s' % (user.id, start, end, key_hash)
    if not created:
        faves = cache.get(key)
        if faves is not None:
            return faves
    faves = Favorite.objects.filter(user=user)[start:end]
    cache.set(key, list(faves), 86400 * 7)
    return faves

As you can see, the first thing we do is grab that hash for the user, then we use it as the last part of the key for the function. The whole if not created thing is just an optimization that helps to avoid cache fetches when we know they will fail. Here's the great thing now: invalidating all of the different cached versions of get_favorite for a given user is a single function call:

from django.core.cache import cache

def clear_favorite_cache(user):
    cache.delete('favorite-list-hash-%s' % (user.id,))

By deleting that single key, the next time get_favorites is called, it will call favorite_list_hash which will result in a cache miss, which will mean it will generate a new unique identifier and stuff it in cache, meaning that all of the keys for get_favorites are instantly different. I think that this is a powerful pattern that allows for coarser-grained caching without really sacrificing much of anything.

There is one aspect of this technique that some people will not like: it leaves old cache keys around taking up memory. I don't consider this a problem because memory is cheap these days and Memcached is generally smart about evicting the least recently used data.

I'm interested though, since I don't see people posting much about nontrivial cache key generation and invalidation. How are you doing this type of thing? Are most people just doing naive caching and calling that good enough?

Several months ago after rolling out the lifestreaming features on this site, I became unhappy with the way that the design scaled with the new feature. Around that time I did a purely-visual refactor (just templates and CSS changed--no code), but it has sat untouched on my hard drive for several months.

Today I ran across that redesign and since it essentially looked complete, I deployed it to the site. I'm still not very happy with the design, but I think that this time it's at least easier to read than the last one, which had become very cluttered and difficult to navigate and read.

Let me know what you think in the comments.

Lately I've really fallen in love with writing utilities whose interface is simply HTTP. By making it accessible via HTTP, it's really easy to write clients that talk to the utility and, if the need arises, there are lots of tools that already exist for doing things with HTTP, like load balancing and caching, etc.

While it would be easy to use a framework to build these utilities, lately I've been choosing not to do so. Web frameworks like Django and Pylons are great when you need to build a fully-featured web application that will be accessible by people. When it will only be computers talking to the service, however, a lot of the machinery provided by frameworks is unneeded and will only slow your utility down. Instead of using a framework, we're going to write a pure WSGI application.

def application(environ, start_response):
    start_response('200 OK', [('content-type', 'text/plain')])
    return ('Hello world!',)

from collections import defaultdict
counts = defaultdict(int)

def application(environ, start_response):
    global counts
    path = environ['PATH_INFO']
    method = environ['REQUEST_METHOD']
    if path.startswith('/song/'):
        song_id = path[6:]
        if method == 'GET':
            start_response('200 OK', [('content-type', 'text/plain')])
            return (str(counts[song_id]),)
        elif method == 'POST':
            counts[song_id] += 1
            start_response('200 OK', [('content-type', 'text/plain')])
            return (str(counts[song_id]),)
        else:
            start_response('405 METHOD NOT ALLOWED', [('content-type', 'text/plain')])
            return ('Method Not Allowed',)
    start_response('404 NOT FOUND', [('content-type', 'text/plain')])
    return ('Not Found',)

from collections import defaultdict
counts = defaultdict(int)

def application(environ, start_response):
    # ... start of app
    if path.startswith('/song/'):
        # ... song-specific logic
    elif path == '/':
        if method == 'GET':
            res = ','.join(['%s=%s' % (k, v) for k, v in counts.iteritems()])
            start_response('200 OK', [('content-type', 'text/plain')])
            return (res,)
        elif method == 'DELETE':
            counts = defaultdict(int)
            start_response('200 OK', [('content-type', 'text/plain')])
            return ('OK',)
        else:
            start_response('405 METHOD NOT ALLOWED', [('content-type', 'text/plain')])
            return ('Method Not Allowed',)
    # ... rest of app

sudo easy_install Spawning

$ spawn -t 0 -p 8000 counter.application

$ curl http://127.0.0.1:8000/song/1
0
$ curl -X POST http://127.0.0.1:8000/song/1
1
$ curl http://127.0.0.1:8000/song/1
1
$ curl -X POST http://127.0.0.1:8000/song/5
1
$ curl -X POST http://127.0.0.1:8000/song/5
2
$ curl http://127.0.0.1:8000/
1=1,5=2
$ curl -X DELETE http://127.0.0.1:8000/
OK

import httplib

class CountClient(object):
    def __init__(self, servers=['127.0.0.1:8000']):
        self.servers = servers

    def _get_server(self, song_id):
        return self.servers[song_id % len(self.servers)]

    def _song_request(self, song_id, method):
        conn = httplib.HTTPConnection(self._get_server(song_id))
        conn.request(method, '/song/%s' % (song_id,))
        resp = conn.getresponse()
        play_count = int(resp.read())
        conn.close()
        return play_count

    def get_play_count(self, song_id):
        return self._song_request(song_id, 'GET')

    def increment_play_count(self, song_id):
        return self._song_request(song_id, 'POST')

    def get_all_play_counts(self):
        dct = {}
        for server in self.servers:
            conn = httplib.HTTPConnection(server)
            conn.request('GET', '/')
            counts = conn.getresponse().read()
            conn.close()
            if not counts:
                continue
            dct.update(dict([map(int, pair.split('=')) for pair in counts.split(',')]))
        return dct

    def reset_all_play_counts(self):
        status = True
        for server in self.servers:
            conn = httplib.HTTPConnection(server)
            conn.request('DELETE', '/')
            resp = conn.getresponse().read()
            if resp != 'OK':
                status = False
            conn.close()
        return status

>>> from countclient import CountClient
>>> c = CountClient()
>>> c.get_play_count(1)
0
>>> c.increment_play_count(1)
1
>>> c.increment_play_count(1)
2
>>> c.get_play_count(1)
2
>>> c.increment_play_count(5)
1
>>> c.get_all_play_counts()
{1: 2, 5: 1}
>>> c.reset_all_play_counts()
True
>>> c.get_all_play_counts()
{}

e:Desktop ericflo$ ab -n 10000 http://127.0.0.1:8000/song/1
...
Concurrency Level:      1
Time taken for tests:   7.792 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1020000 bytes
HTML transferred:       10000 bytes
Requests per second:    1283.31 [#/sec] (mean)
Time per request:       0.779 [ms] (mean)
Time per request:       0.779 [ms] (mean, across all concurrent requests)
Transfer rate:          127.83 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       2
Processing:     0    1   0.8      1      43
Waiting:        0    1   0.5      0      43
Total:          1    1   0.8      1      43

Yesterday was the day of "2008 in review" posts. I really enjoyed reading over what people had accomplished during the year, and what they planned on doing in the new year. I hadn't planned on writing this post, but for some reason I'm doing it anyway. So here goes, not much technical stuff ahead, so if personal stuff bores you then click away now.

This post is my final post in the blog-post-per-day challenge. There have been days when I really didn't want to blog (resulting in posts like this), and there have been days where I was excited about what I was writing about and spent a lot of time on a post (resulting in posts like this one). It was much more difficult and time-consuming than I was expecting. Another thing I wasn't expecting which actually kept motivating me to write more: actual traffic to this site. Previously this site saw maybe 100 hits/day due to mostly various google hits on certain Django topics. When I started doing the blog post-per-day thing, however, this is what my traffic turned into:

But as I started to look at where that traffic was coming from, I realized it wasn't organic at all. (insert sad puppy face here) In fact, most of the traffic was coming from just a few sites. Here's what my top 10 referring sites were:

Once I found out that nearly all my traffic was coming from reddit, I started to try to cater towards that audience. To reddit's credit, however, the more I tried to target content towards what I thought redditers would like, the less successful the articles did over there. Either I was doing a bad job of writing articles for that audience, or they wisened up to my act (I'm thinking the latter is more likely).

I was also very surprised by which articles turned out to be the most popular. Here's the list of articles that I thought were my best:

1. Easy Multi-Database Support for Django
2. Writing an Markov-Chain IRC Bot with Twisted and Python
3. Lambda Calculus
4. Drop-dead simple Django caching
5. Using CouchDB with Django
6. Why CouchDB Rocks
7. "Web Hooks"
8. Reverse HTTP

Here's the list of my top 8 most popular articles over the past few days, in order, by traffic:

1. Gems of Python
2. Why CouchDB Sucks
3. It's caches all the way down
4. The internet is in immediate danger of collapse
5. Why use VARCHAR when you can use TEXT?
6. Using CouchDB with Django
7. Why CouchDB Rocks
8. Secrets of the Django ORM

Interestingly enough there are only two posts that appear on both lists. Really, it surprised me which articles got picked up and which didn't. I suppose it has something to do with the sensational titles and the sometimes-controversial posts. Specifically the VARCHAR/TEXT post got much much more push-back than I was expecting. In hindsight, I was wrong to mention anything other than PostgreSQL and SQLite, as those are what I've actually done the TEXT-only experimentation in.

After doing 14 screencasts and now 30 blog posts over the past few months, I'm pretty well spent in terms of creating new, original, content. That's not to say that I'm going to stop writing here or anything like that, but certainly I won't be posting quite as much. When I do, it will be because I legitimately have something to say, instead of because of an obligation.

Thanks for visiting my site while I participated in this challenge. I hope you stick around.

Last week I wrote an article called Why CouchDB Sucks, which many people correctly said should have been called "What CouchDB Sucks at Doing". Nearly everyone pointed out that it was not designed to do the things that I was mentioning in the article. This time around, I'd like to focus on some of the features about CouchDB that I think absolutely rock.

Caching is easy to screw up. Usually it's a manual process which is error-prone and tedious. It's actually quite easy to cache, but knowing when to invalidate which caches becomes a lot harder. There is a subset of caching the caching problem that, with Django, can be done quite easily. The underlying idea is that every Django model has a primary key, which makes for an excellent key to a cache. Using this basic idea, we can cover a fairly large use case for caching, automatically, in a much more deterministic way. Let's begin.

First, we need to decide upon a setting for how long each individual item should be saved in the cache. I'm going to call that SIMPLE_CACHE_SECONDS and grab it like so:

from django.conf import settings

SIMPLE_CACHE_SECONDS = getattr(settings, 'SIMPLE_CACHE_SECONDS', 2592000)

The next thing we need to do is be able to generate a cache key from an instance of a model. Thanks to Django's _meta information, we can get the app label and model name, plus the primary key, and we're all set.

def key_from_instance(instance):
    opts = instance._meta
    return '%s.%s:%s' % (opts.app_label, opts.module_name, instance.pk)

So now let's start setting the cache! My preferred way to do it is via a signal, but you could do it in a less generic way by overriding save on a model. My signal looks like this:

from django.core.cache import cache
from django.db.models.signals import post_save

def post_save_cache(sender, instance, **kwargs):
    cache.set(key_from_instance(instance), instance, SIMPLE_CACHE_SECONDS)
post_save.connect(post_save_cache)

Now that we're putting items in the cache, we should probably delete them from the cache when the model instance is deleted:

from django.db.models.signals import pre_delete

def pre_delete_uncache(sender, instance, **kwargs):
    cache.delete(key_from_instance(instance))
pre_delete.connect(pre_delete_uncache)

This is all good and well, but right now we don't really have a way to get at that information. Cache is pretty useless if we never use it! Our interface to the database is through the model's QuerySet, so let's make sure that our QuerySet is making good use of our newly-populated cache. To do so, we'll subclass QuerySet:

from django.db.models.query import QuerySet

class SimpleCacheQuerySet(QuerySet):
    def filter(self, *args, **kwargs):
        pk = None
        for val in ('pk', 'pk__exact', 'id', 'id__exact'):
            if val in kwargs:
                pk = kwargs[val]
                break
        if pk is not None:
            opts = self.model._meta
            key = '%s.%s:%s' % (opts.app_label, opts.module_name, pk)
            obj = cache.get(key)
            if obj is not None:
                self._result_cache = [obj]
        return super(SimpleCacheQuerySet, self).filter(*args, **kwargs)

The only method that we really need to overwrite is filter, since get and get_or_create both just rely on filter anyway. The first for loop in the filter method just checks to see if there is a query by id or pk, and if so, then we construct a key and try to fetch it from the cache. If we found the item in the cache, then we place it into Django's internal result cache. At that point we're as good as done. Then we just let Django do the rest!

This SimpleCacheQuerySet won't be used all on its own though, we need to actually force a model to use it. How do we do that? We create a manager:

from django.db import models

class SimpleCacheManager(models.Manager):
    def get_query_set(self):
        return SimpleCacheQuerySet(self.model)

Now that we have this transparent caching library set up, we can go around to all of our models and import it and attach it as needed. Here's how that might look:

from django.db import models
from django_simplecache import SimpleCacheManager

class BlogPost(models.Model):
    title = models.TextField()
    body = models.TextField()

    objects = SimpleCacheManager()

That's it! Just by attaching this manager to our model we're getting all the benefits of per-object caching right away. Of course, this isn't comprehensive. It does hit the vast majority of use cases, though. If you were to use this for a real site, however, then you wouldn't be able to use update method. It's a little bit trickier since there's no post_update signal, but it's nowhere near impossible. Let's just say that, for now, it's being left unimplemented as an exercise for the reader. in_bulk would be actually quite fun to implement, too, because you could get all of the results possible from cache, and all the rest could be gotten from the database, then merge those two dictionaries before returning.

I think this would be a really good reusable Django application. Essentially, we've grown a library from the ground up that really isn't all that much code. I think it took me 20 minutes to write the actual code, but with some serious polish and love, this library could evolve into something that I think many reusable apps would use to great benefit. What do you think? What should a good, simple, Django caching library have?

Yesterday I wrote about Web Hooks and how powerful it could be if one web service sends HTTP requests to another web service. Today I want to take that concept one step further. What if you tell that service that you would like it to send a POST request back to you, whenever an event happens? This slight modification makes for a very powerful tool.

Let's take the example of popular real-time web applications like Facebook's instant messenger or FriendFeed's "Real-time" view. Both of these services make use of a technique called long polling, where the client sends an HTTP request and the server does not respond until it has some event to deliver. The client can only keep the request open for so long, so it periodically times out and re-sends the request. (It also re-sends the request if it does receive some data).

The problem with this technique is that it's really trying to turn a client into a server. It's really fighting against the way that HTTP wants to work. So why fight it? Imagine that all of our browsers have simple, lightweight, HTTP servers installed. The client could request to upgrade to reverse HTTP, and then the server could initiate a connection with the client. Now, as events come in to the web service, the service could directly send those updates to the client.

Going back to the example of Facebook IM, here's how that would work: When I open a Facebook page, my client sends a request to Facebook's IM server. Facebook's IM server sends a response with the HTTP/1.1 Upgrade header reading "PTTH/0.9" (funny, huh?). Then, the client knows to accept an HTTP connection from Facebook's IM server. Facebook's IM server then opens that connection with the client, and sends HTTP POSTs every time it receives a new instant message that the client should receive. The client's web browser would have some JavaScript hooks to parse the body of those requests, so that it could update the content of the instant message window on the page.

Isn't this brilliant? It directly meshes with the HTTP protocol, and makes this system which seems like a hack right now, instantly become an elegant solution. I really wish I could take credit for thinking this up, but I did not. My coworker Donovan Preston blew my mind with this a few weeks back. If you're looking for a more visual example of how this might work, or a reference implementation of the protocol in action, check out this wiki page.

A few months back GitHub rolled out its implementation of something that they call "Service Hooks". The idea behind these hooks is that when you commit some new piece of code to GitHub, they want to be able to alert other services that you have committed that code. For example, one of the service hooks is the ability to send a tweet to Twitter, and another of those hooks updates the Lighthouse ticket tracker.

I thought this was a really good idea when they rolled it out, so I did a bit of searching and found out that there is a larger body of work surrounding this idea, and that body of work is called Web Hooks. The central idea behind web hooks is that a user supplies a service that they use with a URL. Then, when that user performs an action on that service, the service agrees to send an HTTP POST request to the user's specified URL, with some information about the action that the user took on the service.

SlideShare has an excellent presentation deck about this idea, which likens it to Unix pipes. That analogy makes a lot of sense if you think about it. With the standard model that most websites follow today, a client can only send requests. This means repeated polling until the client receives the information that it is interested in. With web hooks, however, the service is responsible for passing that action along to the next service. This simple yet powerful mechanism can allow for very advanced systems which interoperate very simply through chaining.

Let's expore a concrete example of what this might look like. A few months back I signed up for a pro account on Flickr, so that I could upload some of the pictures that I had stored on my computer. What I did was to upload some pictures with descriptions, and then I went and posted on Twitter some of the links to those pictures. I also went and added that new Flickr account to FriendFeed so that others could see my pictures as well.

This was all a manual process. If both Flickr and Twitter supported web hooks, I could have simply set up their respective URLs and uploaded my pictures. The process might have happened like this: First, the pictures are uploaded. Then Flickr sends a POST request to Twitter, with the description of the picture and a link to the picture. Twitter sends a POST request to FriendFeed, adding the new item to my FriendFeed lifestream.

You could even write custom scripts to handle the web hooks. For example let's say that I want any tweet with the name 'Kevin' to be sent to my brother's email address. I could add a URL to Twitter linking to a script on my computer which scans the contents of the tweet. If the tweet has the name 'Kevin' in it, it would send an email. If not, it might do nothing.

I think that this concept is very powerful not only in terms of rendering trivial the interoperability between disparate services, but also in terms of simply saving on bandwidth and computing power. Technologies which constantly poll resources hoping for updated content seem silly in comparison to the powerful simplicity that web hooks provide.

There are definitely some drawbacks to a system like this. Firstly, the name: I actually can't think of a worse name for this concept. Web hooks?! Let's come up with something better. All joking aside though, this type of system does face a serious problem when it comes to the question of reliability. If a script receives no POST, it could mean that either no event happened, or that the internet connection went down for a bit, or that the service is down, or any number of other possible things. I think the solution for this is a hybrid model of sparse polling in conjunction with web hooks.

Most of all, this technology just seems so underused. There are ridiculously few people who implement something like this, yet it seems like an undeniably useful service--especially given its relative simplicity to implement. Let's all try to encourage the services that we use on a daily basis to support web hooks, because by doing just that, we can make the web a lot better.

I love screencasts. I love them at least enough to create 16 of them this year. That's why when James Tauber announced that the Pinax Project would be having a screencast competition, it made me very very happy!

Pinax is already a great aide for those who want to build a website with a great deal of functionality in a short period of time. If you haven't checked it out yet, I really hope that you do. It tries to take the best of breed Django applications and glues them together with minimal configurable code and a set of common templates and conventions.

One area that Pinax could be better about, however, is documentation. We're working on it, but it still has a bit to go. That's one of the reasons why this contest is so great. By entering a screencast into the contest, you are really helping the Pinax community out a lot.

The best part about this contest, however, is that there are prizes! One prize, $100 at Amazon, will go to the very best screencast. A second prize, $40 at Amazon, will go to the second best screencast. As of this writing, there are no entries. So what are you waiting for? Head on over to the contest details page and enter your screencast today!

I've never tried to start a meme before, but since I don't really have anything great to post about, I'd like to try to start one. I think this one could be particularly fun because it encourages implementation of a basic program in many different languages.

Rules:

1. Implement a program that takes in a user's name and their age, and prints hello to them once for every year that they have been alive.
2. Post these rules, the source code for your solution, and the following list (with you included) on your blog.
3. Bonus points if you implement it in a language not yet seen on the following list!

The List:

1. [Python] http://www.eflorenzano.com/blog/post/trying-start-programming-meme
2. [Bash] http://aartemenko.com/texts/bash-meme/
3. [C] http://dakrauth.com/media/site/text/hello.c
4. [Java] http://adoleo.com/blog/2008/nov/25/programming-meme/
5. [Python 3] http://mikewatkins.ca/2008/11/25/hello-meme/
6. [Ruby] http://stroky.l.googlepages.com/gem
7. [Ruby] http://im.camronflanders.com/archive/meme/
8. [Lisp] http://justinlilly.com/blog/2008/nov/25/back-on-the-horse/
9. [Lua] http://aartemenko.com/texts/lua-hello-meme/
10. [Functional Python] http://aartemenko.com/texts/python-functional-hello-meme/
11. [Erlang] http://surfacedepth.blogspot.com/2008/11/erics-programming-meme-in-erlang.html
12. [Haskell] http://jasonwalsh.us/meme.html
13. [PHP] http://fitzgeraldsteele.wordpress.com/2008/11/25/memeing-in-php-2/
14. [Javascript] http://www.taylanpince.com/blog/posts/responding-to-a-programming-meme/
15. [Single-File Django] http://www.pocketuniverse.ca/archive/2008/november/27/florenzano-factor/

(...and make sure to check out the comments on this post for some other fun implementations!)

And here's my implementation:

name = raw_input('Please enter your name: ')
age = int(raw_input('Please enter your age: '))

for i in xrange(age):
    print "%2d) Hello, %s" % (i, name)

And its output:

Please enter your name: Eric Florenzano
Please enter your age: 22
 0) Hello, Eric Florenzano
 1) Hello, Eric Florenzano
 2) Hello, Eric Florenzano
 3) Hello, Eric Florenzano
 4) Hello, Eric Florenzano
 5) Hello, Eric Florenzano
 6) Hello, Eric Florenzano
 7) Hello, Eric Florenzano
 8) Hello, Eric Florenzano
 9) Hello, Eric Florenzano
10) Hello, Eric Florenzano
11) Hello, Eric Florenzano
12) Hello, Eric Florenzano
13) Hello, Eric Florenzano
14) Hello, Eric Florenzano
15) Hello, Eric Florenzano
16) Hello, Eric Florenzano
17) Hello, Eric Florenzano
18) Hello, Eric Florenzano
19) Hello, Eric Florenzano
20) Hello, Eric Florenzano
21) Hello, Eric Florenzano

Well there you have it, my attempt at a programming meme. I hope someone enjoys it!

CouchDB really sucks at doing some things. That should come as no surprise, as every technology has its advantages and its drawbacks. The thing is, when a new technology comes out that looks really promising and cool, everyone writes about all of its advantages, and none of its drawbacks. Then, people start to use it for things it isn't very good at, and they are disappointed. In that spirit, I would like to talk about some of the things that (in my experience) CouchDB is absolutely not good at, and that you shouldn't try to use it for.

First, it doesn't support transactions in the way that most people typically think about them. That means, enforcing uniqueness of one field across all documents is not safe. A classic example of this would be enforcing that a username is unique. You can check whether a username exists, and if not, create a new one. There is no guarantee, however, that between the time that your app has checked for its existence, and the time that you write the new user to the database, that some other instance of your app hasn't beat you to that write.

Another consequence of CouchDB's inability to support the typical notion of a transaction is that things like inc/decrementing a value and saving it back are also dangerous. Fortunately there aren't many instances that you would want to simply inc/decrement some value where you couldn't just store the individual documents separately and aggregate them with a view.

Secondly, CouchDB sucks at dealing with relational data. If your data makes a lot of sense to be in 3rd normal form, and you try to follow that form in CouchDB, you're going to run into a lot of trouble. Yes, it's probably possible with tricks with view collations, but you're constantly going to be fighting with the system. If your data can be reformatted to be much more denormalized, then CouchDB will work fine.

Thirdly, CouchDB sucks at being a data warehouse. In every data warehouse that I've ever run into, people have all kinds of different requests for how to slice the data. And they all want it to be done, yesterday. The problem with this is that temporary views in CouchDB on large datasets are really slow, because it can't use any of its normal indexing tricks. If you by some chance have a very rigid way of looking at your data, using CouchDB and permanent views could work quite well. But in 99% of cases, a Column-Oriented Database of some sort is a much better tool for the data warehousing job.

So does CouchDB suck? No, it's by far my favorite new database technology on the block. What it's good at doing, it's great at doing, but that doesn't mean that it should be used for everything. With the kinds of scaling issues that we're seeing with today's highly-interactive web applications, we need to make use of a broad range of technologies, and use each one for its greatest strengths. That's called using the right tool for the job, and that's never gone out of style.

A few months back, I had a bit of an epiphany. I suspect I was about 5 years too late with this epiphany, but nevertheless here it is: Microsoft, Sun, and Adobe all have these virtual machines that they want everyone to develop on top of. For Microsoft it's the CLR, for Sun it's the JVM, and for Adobe it's AVM. The problem that each of them have is that not everyone goes out of their way to install all of these VMs, so each vendor only has a subset of the entire computing space.

My epiphany was that there is one VM which nearly every modern computer has access to. It's JavaScript. Every browser developed in the past 10 years ships with some variation of it, and some systems even come with it as a command-line option. Whilst they do have a big problem in terms of standardization (or lack thereof), there is certainly a lowest common denominator that could be useful for writing apps on top of.

That being the case, why don't we see more language implementations on top of JavaScript? There is Processing.js, a port of the Processing programming language. There is also Objective J and Cappuccino, a port of Objective C and Cocoa, respectively. Each of those language implementations have received quite a bit of attention (both positive and negative, mind you).

So why don't we see Ruby, Python, or other languages implemented on top of JavaScript? For that matter, why don't we see more ports of apps to JavaScript? I know that in the Python world, PyPy has a JavaScript backend, but to the best of my knowledge that backend has been all-but-abandoned. I think it would be really cool if we were to see more applications and programming languages targeting the most widely adopted virtual machine ever: JavaScript.

It seems that the world of computing is always oscillating between offloading more work to the server, and offloading more work to the client. In the very early days of dumb terminals, we had most of the actual computation being done on big mainframes. Then personal computers became more powerful, allowing for much richer applications to be created.

With the advent of the internet, the landscape of computing architectures shifted again to being done focused on the server. Now, however, JavaScript has become more powerful. Combined with things like SVG and the canvas tag, we can create extremely rich applications that take place solely in the browser.

Projects like CouchDB are even starting to open the door for truly peer-to-peer web applications. With all of this taking place, it seems that client-side web applications are poised to see some fairly strong growth. This is especially evident now that companies as big as Google seem interested in the idea, with its Google Gears product that allows you to "work offline". But there are certain things that need to be satisfied first.

We need a way of enforcing security across these apps. It looks like some combination of OpenID and OAuth are going to be the winners in this space, but I've never seen a seamless implementation of either of these protocols, even by the companies most invested in the technology. There is a lot of work to go on usability before authentication and authorization are ubiquitous through these open protocols.

We also need to standardize more on the data interchange formats that we use to shuttle information back and forth between these different apps. Atom goes a long way towards describing the data that we use, but its adoption is nowhere near ubiquitous, and some sites still rely on older, more outdated, RSS syndication formats that aren't quite up to the task.

But even if we standardize on some application platform (be it Google Gears or CouchDB or some other container), security, and data interchange formats, there are certain things that need to be considered. For one, there are some applications that just aren't practical to be implemented on the client. Video editing comes to mind (and I would know, considering I interned for eyespot, a company which was attempting to do just that).

Another concern is that, as we've seen with the emergence of standards for CSS and HTML, a certain amount of rigidness is good, but a strict conformist attitude leads to significantly stifled innovation. If you were to write an app that doesn't fall within the boundaries of what's possible given the agreed-upon standards, would you still be able to go forward with the development of the app, or would you run into resistance from those who have a stake in those standards?

In all, I have a feeling that we are going to move more and more to a hybrid approach, with much more logic being computed on the side of the client (especially in terms of visual components and interactivity), and that much more of the server side is going to be involved in slicing and serving up just the raw data. We can see this happening today with technologies like AJAX being touted as the centerpiece of some "Web Two Point Oh" sites. I'm excited to see where this will all go, and more than excited that, being a developer during this time, get to help shape that direction.

I posted a while back about using CouchDB with Django, talking about how one might build a lifestreaming application combining the flexibility and power of CouchDB with the ease of use and utility of Django. I even linked to the project on github. At the time I was taking a hard look at the Django-based software that was running this blog. It needed some work, to say the least. So I did a bit of cleanup, a LOT of reorganization, and integrated this CouchDB-based lifestreaming application into the mix.

There's no visual refresh--that will come some other time. For right now, it's just a backend refresh, so for the most part nobody will notice anything. But I can rest much easier knowing that the backend is just a little bit less of a mess. It's also a LOT easier to deploy. I've integrated a Fabric deployment script so that all I need to do is type:

fab deploy

And everything gets zipped up, sent to the server, unziped, and all of the relevant processes get restarted. Pretty cool! That's going to allow for much more rapid iteration on the site, so changes can come more often and bugs can be fixed.

Because of that, expect more experimentation. As I'm experimenting with Django stuff, I'll do a lot more public demos where the experimentations are public for the world to see and play around with. I've got lots of ideas for Django-y experiments and it's going to be really fun to see how they shape up.

So what do you think? Is the new lifestreaming stuff annoying or cool?

It seems that people either love things like lambda calculus and eat it up, or their eyes gloss over and they don't care or want to learn about it. I was firmly part of the latter group, until I took a class in college that forced me to learn about it. The thing is, once you learn more about a subject like this, the less you'll understand why you were so averse to learning about it.

In the spirit of broadening our horizons, let's explore the lambda calculus and see if it's as hard as it's cracked up to be. I don't claim to be a wizard at this stuff, and by no means am I a theoretical computer scientist, but that's not the goal. The goal is to cut down on the formality and see if we can expore this stuff a bit. First, let's look at a very basic function: addition. In lambda calculus it looks like this:

>> x. x + 1

That is, a function whose argument is named x, returns the value x + 1. This looks really similar to some syntax that we have in Python. Let's see if we can find a corollary to help make things easier:

lambda x: x + 1

Surprisingly, the syntax is almost the same! It's an anonymous function which takes x as its one argument and returns x + 1. Lambda calculus doesn't have powerful enough syntax for talking about functions with multiple arguments, though, so you may think that it's not very powerful. Thanks to a technique called currying, however, we can express functions with infinite numbers of arguments. This is how that might look:

>> x. >> y. x + y

This is where I think the syntax starts to get in the way of readability. Let's see what this would look like in Python:

lambda x: lambda y: x + y

Hmmm...it's not much more readable, but at least we can open up an interactive interpreter to see what we're doing. Basically what we're doing is creating a function which returns another function. That inner function then has access not only to its single argument, but to its parent's argument as well. Let's demonstrate that this works:

>>> add = lambda x: lambda y: x + y
>>> add(5)(4)
9

Now does this work if we start to change the variable names? What if I just decide that I can't stand the letter x and don't want it inside any more of my functions. Well I can simply change the variable name, as long as I change any uses of that variable. It would work just the same:

>>> add = lambda z: lambda y: z + y
>>> add(5)(4)
9

In this instance, we have changed all of our instances of x with z. This ability to change the names of bound variables is the first rule of lambda calculus, and it's called ?-conversion.

As developers, we intuitively understand the idea of function application. We understand that a function which takes in an argument replaces all references to that argument with the given value, and returns a result. For example, in the addition function, lets go through the steps for function application:

>>> add = lambda x: lambda y: x + y
>>> add5 = add(5)
# We can envision what's happening here by replacing x with 5
# add5 = lambda 5: lambda y: 5 + y
# add5 = lambda y: 5 + y
>>> add5(4)
# And now the final function application, resulting in a value
# lambda 4: 5 + 4
9

This idea of function application, in lambda calculus is given the name 2-reduction. To review, in lambda calculus syntax that process would look like so:

(>> x. >> y. x + y) 5
(>> y. 5 + y) 4
5 + 4
9

Something that we also intuitively know as developers is that functions can do the exact thing, but have very different implementations. For example this function:

>> x. x + x

...will give the same results for all values as this function:

>> x. x * 2

We understand that code which uses the former function can easily swap out the latter and expect the program the function correctly. This idea is called .-conversion in lambda calculus.

See, this is pretty simple stuff! Obviously there are subtleties that I didn't go into, and it gets a bit more confusing when we start to try to figure out formally which variables occur bound and which occur free, and as we attempt to preserve that status while doing ?-conversion, 2-reduction, and .-conversion.

Here's some really mind bending food for thought:

(>> x. x x) (>> x. x x)

Think about how this would reduce:

(>> x. x x) (>> x. x x)
(>> (>> x. x x). (>> x. x x) (>> x. x x))
(>> x. x x) (>> x. x x)

As you can see, we could do this forever. This is called the ? combinator, and it lets us do some really cool things with recursion. Maybe more on that in a later blog post.

What is all of this good for? Well, with only this very simple, rigidly defined ruleset, we can express every possible computer program. That makes it very good for doing mathematical proofs and other various scholarly things when we want to explore algorithms. It also forms the basis of all programming languages, and especially functional programming languages.

Of particular interest to me is that we can actually represent numbers, truth, and logic using only these very basic primitives. I'll explore that in another post. To me, all of this is fascinating, and it's hard to believe that before my teacher forced me to learn it, I actively didn't want to know about it.

Whenever I tell people that I'm participating in this blog post per day event, most people begin laughing hysterically. Then the next thing that people typically do, unprompted, is to start making excuses as to why they don't blog. Most of the time it is something like, "I don't feel like I have anything important to say," and sometimes it's more like "I tried it but nobody was reading it, so I gave up." These are valid reasons why someone shouldn't blog.

But you shouldn't make those excuses, and you should blog anyway. Firstly, writing is a valuable skill in any profession. Simply forcing yourself to write more helps to improve your writing abilities, and ensures that you're able to organize your thoughts in a way that others can understand. Even if it doesn't make your writing any better, it will make you more aware of your shortcomings as a writer. Myself, for instance: I have found that a big problem with my writing is that I start too many sentences with conjugations. I also use way too much of the passive voice and alternate between subjective personal pronouns.

Another reason why you should blog is because it forces you to look at things from different perspectives. I can't count how many times I've begun writing an opinion post, and give up after a while because after putting it into writing, I realize how weak my actual argument is. It makes me question some of my opinions. Conversely, if the argument comes across as strong and obvious, then I am even more certain of my opinion.

Finally, blogging is a classic chicken-and-egg problem. If you don't blog, nobody will read your blog. If people don't read your blog, you won't want to blog. Fortunately if you just remain steadfast during the beginning portion of your blogging life (and make sure to participate with other bloggers who talk about similar things as you), then people will surely start to read your posts.

Honestly now that I'm reading through this article, it seems a bit weak. It's 11:15PM and I can't think of another subject to write about before midnight, though, so this post will have to do!

I hope that you guy start/continue to blog, because it really is a good habit to get into.

Most people in the Django community are deploying their apps these days with mod_wsgi. If not, then you're at least using WSGI as a communication layer with your application server, in one way or another. The great thing about WSGI is that it gives everyone a common interface through which to talk. It also has the added benefit of being a common abstraction that many people have built these great, really useful tools on top of.

Consider Repoze. If you navigate to the middleware section of their website, they have some really cool stuff available! There are utilities for logging, authentication, security, profiling, templating, etc. All of these pieces of middleware are designed to be totally pluggable, because they are designed to work solely based on what's available through WSGI.

My personal favorite of that lot is repoze.profile. It accumulates Python profiling information about whatever app is being run, and allows you to view that profile information via a web interface by visiting a special URL. There is absolutely no reason that the Pylons, TurboGears, or CherryPy guys should be able to get away with keeping this stuff for themselves, so I want to show just how easy it is to integrate this profiling module with Django.

First, though, here's a typical .wsgi file that might be used in conjunction with mod_wsgi:

import os, sys
sys.stdout = sys.stderr

os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

import django.core.handlers.wsgi

application = django.core.handlers.wsgi.WSGIHandler()

There's really nothing special going on here, and if you would like to learn more about how to set up this WSGI file, visit mod_wsgi's documentation on the subject. Now if you'll notice, application is simply an instance of WSGIHandler, which is simply a callable. A WSGI middleware is just a wrapper around that callable. Here's how easy it is to add the profiling middleware:

from repoze.profile.profiler import AccumulatingProfileMiddleware
application = AccumulatingProfileMiddleware(
    application,
    log_filename='/tmp/djangoprofile.log',
    discard_first_request=True,
    flush_at_shutdown=True,
    path='/__profile__')

There we go! We have imported the profiling middleware, and passed the Django WSGI application as the first argument. The rest is just setting options for the middleware. You can restart apache and the WSGI profiling middleware is already working.

Sometimes, though, you don't want all of Apache just to run some middleware. You want to be able to do the same thing, but locally. Believe it or not, Django's local development server is just a WSGI server itself, so one option would be to do the wrapping directly in django, right here. But you really don't want to be hacking inside of Django internals if you don't have to. Fortunately there are many alternative WSGI servers out there. Brian Rosner has created a custom management command to use the excellent CherryPy WSGI server with Django, on his blog.

Let's say you just want to try this out quickly after reading this blog post, though. If you're running Python 2.5 or greater, you're in luck, because a script less than 10 lines long can get you up and running:

#!/usr/bin/env python

import sys
from wsgiref.simple_server import make_server

if __name__ == "__main__":
    execfile(sys.argv[1])
    httpd = make_server('', 8000, application)
    httpd.serve_forever()

Now, to run it, simply invoke it like this:

python runserver.py my_wsgi_file.wsgi

Now, navigate around your app for a little bit and then point your browser to the profile url and see how freaking awesome middleware can be.

I'm not trying to stir up any controversy, I'm not saying we should stop making Django middleware or anything like that. But I seriously, seriously hope that someone tries this out and realizes the multitudes of great WSGI apps out there that can be taken advantage of. Mark Ramm wasn't full of hot air when he talked about this at DjangoCon or blogged about it later. He was right, and I for one wish I had listened sooner.

Twisted is one of Python's great secret weapons. It is an absolute workhorse, allowing for insanely fast network applications to be written with very little effort. So let's do what everyone does when they want to learn more about Twisted: let's write an IRC bot! This bot is going to use Markov Chains to simulate human speech. For whatever reason, I named this bot "YourMomDotCom".

First let's create a skeleton on top of which the rest of the bot will be created:

from twisted.words.protocols import irc
from twisted.internet import protocol

class MomBot(irc.IRCClient):
    def _get_nickname(self):
        return self.factory.nickname
    nickname = property(_get_nickname)

    def signedOn(self):
        self.join(self.factory.channel)
        print "Signed on as %s." % (self.nickname,)

    def joined(self, channel):
        print "Joined %s." % (channel,)

    def privmsg(self, user, channel, msg):
        print msg

class MomBotFactory(protocol.ClientFactory):
    protocol = MomBot

    def __init__(self, channel, nickname='YourMomDotCom'):
        self.channel = channel
        self.nickname = nickname

    def clientConnectionLost(self, connector, reason):
        print "Lost connection (%s), reconnecting." % (reason,)
        connector.connect()

    def clientConnectionFailed(self, connector, reason):
        print "Could not connect: %s" % (reason,)

We've now created an IRCClient subclass which will hold our application logic, and we've also written a factory class which will create instances of that MomBot client. Let's tie these together and start the event loop:

import sys
from twisted.internet import reactor

if __name__ == "__main__":
    chan = sys.argv[1]
    reactor.connectTCP('irc.freenode.net', 6667, MomBotFactory('#' + chan))
    reactor.run()

Now already we have a complete working IRC bot. Right now all it will do is connect to an IRC channel and echo all of the output to the command line. Not bad for how little code we've written. Now all we have to do is implement our application logic. Let's first start by creating the 'brain' of our Markov chain responder, and adding a function to train the brain:

from collections import defaultdict

markov = defaultdict(list)
STOP_WORD = "\n"

def add_to_brain(msg, chain_length, write_to_file=False):
    if write_to_file:
        f = open('training_text.txt', 'a')
        f.write(msg + '\n')
        f.close()
    buf = [STOP_WORD] * chain_length
    for word in msg.split():
        markov[tuple(buf)].append(word)
        del buf[0]
        buf.append(word)
    markov[tuple(buf)].append(STOP_WORD)

In this, we are creating a defaultdict of lists. For every n-word sliding window, the word after that window is appended to the list of possible words. Here's an image which hopefully depicts better than words how the algorithm populates the brain:

But what good is a brain like this if we can't get words back from it. We're going to need to write a function to generate sentences from that brain:

def generate_sentence(msg, chain_length, max_words=10000):
    buf = msg.split()[:chain_length]
    if len(msg.split()) > chain_length:
        message = buf[:]
    else:
        message = []
        for i in xrange(chain_length):
            message.append(random.choice(markov[random.choice(markov.keys())]))
    for i in xrange(max_words):
        try:
            next_word = random.choice(markov[tuple(buf)])
        except IndexError:
            continue
        if next_word == STOP_WORD:
            break
        message.append(next_word)
        del buf[0]
        buf.append(next_word)
    return ' '.join(message)

We start out our seed buffer with the first few words of the message, and if the message wasn't long enough, we fill the seed buffer with some random words from the markov's brain. Then we use the buffer as a key into the markov brain and randomly pick one of the values as our next word. Then we slide that buffer so that the chosen word is now the next word in the buffer (ejecting the oldest word in the buffer). If we ever see a stop word, we stop and return the generated sentence.

Now it's a matter of expanding our bot to take advantage of our markov brain. The first change we will need to make is to modify the MomBotFactory to take more parameters in its __init__ method:

class MomBotFactory(protocol.ClientFactory):
    protocol = MomBot

    def __init__(self, channel, nickname='YourMomDotCom', chain_length=3,
        chattiness=1.0, max_words=10000):
        self.channel = channel
        self.nickname = nickname
        self.chain_length = chain_length
        self.chattiness = chattiness
        self.max_words = max_words

    # ...

Now that it has all of that new information, we can add the final bit of functionality to MomBot responding using sentences generated from the markov brain by updating the privmsg method:

class MomBot(irc.IRCClient):

    # ...

    def privmsg(self, user, channel, msg):
        if not user:
            return
        if self.nickname in msg:
            msg = re.compile(self.nickname + "[:,]* ?", re.I).sub('', msg)
            prefix = "%s: " % (user.split('!', 1)[0], )
        else:
            prefix = ''
        add_to_brain(msg, self.factory.chain_length, write_to_file=True)
        if prefix or random.random() <= self.factory.chattiness:
            sentence = generate_sentence(msg, self.factory.chain_length,
                self.factory.max_words)
            if sentence:
                self.msg(self.factory.channel, prefix + sentence)

If it gets a message from a non-user, we don't care about it, so we just disregard that message. Otherwise, if it's a message directed at the bot, we want to respond to the sending user so we add the user as a prefix to our next message. Then we grab a sentence from the markov brain and send it to the channel.

Not so bad. Now lets modify our startup portion of the script to re-train the brain with anything learned during the course of the bot's life. Here's how the new script will look:

if __name__ == "__main__":
    try:
        chan = sys.argv[1]
    except IndexError:
        print "Please specify a channel name."
        print "Example:"
        print "  python yourmomdotcom.py django-hotclub"
    if os.path.exists('training_text.txt'):
        f = open('training_text.txt', 'r')
        for line in f:
            add_to_brain(line, chain_length)
        print 'Brain Reloaded'
        f.close()
    reactor.connectTCP('irc.freenode.net', 6667, MomBotFactory('#' + chan,
        'YourMomDotCom', 2, chattiness=0.05))
    reactor.run()

I have found personally that, counter-intuitively, a length of 2 seems to produce the most realistic-looking responses. We've really accomplished a lot in a very little amount of code. Instead of my normal closing remarks, let me simply close with a snippet of conversation recently had with YourMomDotCom:

[9:57pm] ericflo: YourMomDotCom: Hello, I'd like to show you off for the readers of my blog.
[9:57pm] YourMomDotCom: ericflo: Hello, I'd like to do, after checking with another.
[9:57pm] ericflo: YourMomDotCom: I'm not exactly sure what you mean by that.
[9:57pm] YourMomDotCom: ericflo: I'm not sure, but I'm in "Easter mode" already I think.
[9:58pm] ericflo: YourMomDotCom: That's funny since we aren't really very close to easter.
[9:58pm] YourMomDotCom: ericflo: That's funny since i missed much of a ForeignKey field?
[9:58pm] ericflo: YourMomDotCom: I miss ForeignKey fields all the time
[9:58pm] YourMomDotCom: ericflo: I miss that
[9:58pm] ericflo: YourMomDotCom: We both agree, then.
[9:58pm] YourMomDotCom: ericflo: We both agree, then.
[9:58pm] ericflo: YourMomDotCom: yay!

Today I found myself at Super Happy Dev House, amongst a whole house full of geeks (I say this in the most friendly way possible). I thought for a while about working on Pinax project, which is what I really wanted to do. However, it didn't seem geeky enough for the occasion.

No, today the call of the geek was too strong. There was only one place to turn to achieve maximal geekiness: functional programming. Not only functional programming, but I thought this occasion called for a concurrency-oriented programming language. Yes, today was the perfect day for some Erlang.

I've done a few small projects with Erlang, but never before have I used OTP. So the goal is to write an OTP gen_server which would store a user's "presence" (think Twitter, Pownce, etc.) in memory. No, it's not a good idea. Yes, it's kind of fun. So here we go!

First we need to decide on an API:

start_link() ->
    gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).

post_presence(UserID, Presence) ->
    gen_server:cast(?SERVER, {post_presence, UserID, Presence}).

list_presence(UserID) ->
    gen_server:call(?SERVER, {list_presence, UserID}).

list_presence(UserID, Limit) ->
    gen_server:call(?SERVER, {list_presence, UserID, Limit}).

public_list(Limit) ->
    gen_server:call(?SERVER, {public_list, Limit}).

start_link will start the server. post_presence will post a user's current presence to the server. list_presence with one argument gets all of the specified user's presence information. With a second argument, it limits the amount of returned presence information to the specified number. Calling public_list means that you're getting the latest posts from anyone.

Before that, though, let's get some of the gen_server cruft out of the way:

-module(presence_database).
-behavior(gen_server).

-define(SERVER, ?MODULE).

%% gen_server API
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2,
    code_change/3]).

%% presence API
-export([start_link/0, post_presence/2, list_presence/1, list_presence/2,
    public_list/1]).

handle_info(_Info, State) ->
    {noreply, State}.

terminate(_Reason, _State) ->
    ok.

code_change(_OldVersion, State, _Extra) ->
    {ok, State}.

init([]) ->
    {ok, dict:new()}.

These are mostly things that we need to have in order for things to work. We define our module name, the behavior that this module mimics, and define a constant. Then, we export the required functions for gen_server and export our public presence api functions. Finally we implement really simple functions for those gen_server functions that we don't care much about.

Now since I couldn't figure out how to slice a list in Erlang, I wrote my own slice function. Make sure to flame me for this.

slice(List, Start, End) ->
    slice(List, Start, End, 0, []).

slice([], _Start, _End, _Index, Acc) ->
    lists:reverse(Acc);
slice([Item | Rest], Start, End, Index, Acc) ->
    case Index >= Start of
        true ->
            case Index < End of
                true ->
                    slice(Rest, Start, End, Index + 1, [Item | Acc]);
                false ->
                    slice(Rest, Start, End, Index + 1, Acc)
            end;
        _ ->
            slice(Rest, Start, End, Index + 1, Acc)
    end.

...and I'm pretty sure that using guard expressions this could be done in one case statement. Or maybe it could be done in a single list comprehension. I don't know. This works for our purposes.

Now we have to write functions called handle_cast and handle_call. handle_cast is essentially a server function which you don't expect a response from. Messages can queue up to this function and they will be handled sequentially but never return any value to the caller. handle_call is exactly the opposite. The caller is expecting a response, so this is a blocking operation.

Let's use handle_cast to accept new presence notifications:

handle_cast({post_presence, UserID, Presence}, State) ->
    case dict:is_key(UserID, State) of
        true ->
            {noreply, dict:append(UserID, {erlang:now(), Presence}, State)};
        _ ->
            {noreply, dict:store(UserID, [{erlang:now(), Presence}], State)}
    end;
handle_cast(_Msg, State) ->
    {noreply, State}.

In essence, we check to see if the user has registered their presence, and if so, we add their presence to their presence list. If they haven't submitted their presence before, we create a new presence list for them and add their submitted presence to that list. We have also generated a catchall version of the function for when the call doesn't match the signature {post_presence, UserID, Presence} for some reason.

Now let's do the harder one: the call to get various presence information from our server:

userfy_list(User, List) ->
    lists:map(fun({Time, Msg}) -> {User, Time, Msg} end, List).

handle_call({list_presence, UserID}, _From, State) ->
    case dict:find(UserID, State) of
        {ok, Value} ->
            {reply, Value, State};
        error ->
            {reply, {error, user_does_not_exist}, State}
    end;
handle_call({list_presence, UserID, Limit}, _From, State) ->
    case dict:find(UserID, State) of
        {ok, Value} ->
            {reply, lists:nthtail(Limit, Value), State};
        error ->
            {reply, {error, user_does_not_exist}, State}
    end;
handle_call({public_list, Limit}, _From, State) ->
    LatestEntries = lists:flatten([userfy_list(User, List) || {User, List} <- dict:to_list(State)]),
    Sorted = lists:sort(fun({_, A, _}, {_, B, _}) -> A > B end, LatestEntries),
    {reply, slice(Sorted, 0, Limit), State};
handle_call(_Request, _From, State) ->
    {reply, ok, State}.

The first version of handle_call is very straightforward. It simply looks up the list of presence information for a given user and returns that in the reply. The second version does a similar thing, but calls lists:nthtail on the value to limit the number of presence data that is included in that list.

The final version of handle_call is the most complicated, because we want to get information about everyone, order it by the date that it was posted, and limit the number of returned results by the specified limit. An added complexity is that we have to change the data format to include the name of the user who posted it.

First it uses a list comprehension to take the state dictionary and run our userfy_list function on every User/List pair. This will add the user to the presence tuple. Then we flatten that list to be just a single list of userfied tuples. Then, we sort that by the middle value (the timestamp of the presence). Finally, we use our newly-created slice function to take just the slice of information that we care about and return it to the user. Again, there is a catchall handle_call which discards incorrectly-formatted messages.

1> c(presence_database).
{ok,presence_database}
2> presence_database:start_link().
{ok,<0.37.0>}
3> presence_database:post_presence("ericflo", "I am at Super Happy Dev House").
ok
4> presence_database:post_presence("ericflo", "I am writing erlang code").
ok
5> presence_database:post_presence("dreid", "I am having fun at SHDH").
ok
6> presence_database:list_presence("ericflo").
[{{1226,816100,955},"I am at Super Happy Dev House"},
 {{1226,816113,249498},"I am writing erlang code"}]
7> presence_database:list_presence("dreid").
[{{1226,816135,937300},"I am having fun at SHDH"}]
8> presence_database:public_list(5).
[{"dreid",{1226,816135,937300},"I am having fun at SHDH"},
 {"ericflo",{1226,816113,249498},"I am writing erlang code"},
 {"ericflo",
  {1226,816100,955},
  "I am at Super Happy Dev House"}]
9> presence_database:public_list(2).
[{"dreid",{1226,816135,937300},"I am having fun at SHDH"},
 {"ericflo",{1226,816113,249498},"I am writing erlang code"}]

A few years ago, when I first started using Python, I spent an inordinate amount of time looking for "a good Python IDE." The classic IDE links page on the Python wiki was around back then, and looked fairly similar to the way it looks now. I was used to Java and Eclipse. Java, if you haven't used it, is difficult at best to use without an IDE.

Understandably, then, I wanted to see what was available (and free). At the time, the two reigning free ones were Eric3, and SPE, and PyDev was quickly gaining popularity. I found Eric3 to be extremely buggy, crashing every few minutes. Everyone seemed to love it, but I couldn't get it to work stably for me, which was a deal breaker.

PyDev seemed too heavyweight. After all, I was trying to switch away from Java! Plus, the fact that it stored metadata in these strange .project files bugged me to no end. SPE was one that I actually used for a while. It was stable, seemed to be featureful, and had a good community around it. But I found that the interface ended up getting in the way more than helping, and eventually I abandoned that as well.

I tried trial versions of Wing and Komodo, and of course they were the best, but they sure didn't seem like they were worth what their respective companies were asking for them. Anyway, at this point several months had gone by, and I had gotten pretty comfortable with Python. The search for an IDE ended up making me proficient enough that I found I didn't need an IDE at all!

Today, I code in TextMate. But I have no strong preference over any other text editor (in fact, I think my preferred text editor is Kate), but the simplicity that it brings is pretty much unbeatable.

All of this brings me to my question for anyone who's willing to answer: should I take a second look at any of these IDEs? A few years after I did this research and I'm willing to entertain the idea that it's possible an IDE could make development easier. Is there a new kid on the block that is sweeping people off their feet?

"Another feature that's easy to add is a per-entry way to allow or disallow comments." From Practical Django Projects by James Bennett.

Meme from James Tauber, Greg Newman, Justin Lilly and Brian Rosner.

• Grab the nearest book.
• Open it to page 56.
• Find the fifth sentence.
• Post the text of the sentence in your journal along with these instructions.
• Don't dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

The MOST ironic part about this all, is that I was not going to do this meme tonight, and instead write a post. However, I was so busy trying to beat James Bennett's score on level 3 of this challenge, that I ran out of time for tonight.

P.S. I dare you to beat my now 357,500 points on level 3 of the game in that link. I bet you can't.

Admit it, you've done it many times before: you create a database schema, arbitrarily guessing at upper limits to the lengths of various columns, only to later have to perform annoying schema upgrades as you change those columns to fit real-world data.

If you're using PostgreSQL, what you're doing is pointless. There is quite literally no performance benefit to be had, and possibly a performance penalty as the database needs to check the length constraint. This fact can even be found right here, in the official documentation.

If you're using MySQL with InnoDB, it's practically the same situation. The data is laid out on disk in exactly the same way for both TEXT and VARCHAR fields, as explained here. I couldn't find any resources about MyISAM other than that TEXT is stored externally, but I just fired up a test table and did some rudimentary benchmarking and the numbers were well within the margin of error.

If you're using SQLite, everything's a TEXT whether you want it to be or not ( with the notable exception of INTEGER PRIMARY KEY) so it doesn't matter what you try to specify, it will be a TEXT.

I'm as guilty as anyone else on this--I use varchar all the time! But come on, let's stop imposing these arbitrary character limits on our columns when the only reason we're doing it is for historical reasons. Is anyone with me?

Way back in the early days of Django, it used to magically import certain functions before any views were ever run. At some point that bit of magic got stripped out and now we have to explicitly import the things that we are going to use...much like everything else in Python. One thing that Simon Willison suggested at DjangoCon is that the core developers might think about including a common set of functions that could be imported and used as shortcuts instead of importing everything again and again.

I shrugged it off at the time, thinking it was a solution in search of a problem, but wanting to keep an open mind, I wanted to try it out in one app to see how it would look. In this spirit, I created a file called dj.py inside of a project of mine, django-avatar. I looked around at what some of the most commonly used functions were, and came up with a dj.py file that looked something like this:

from django.http import HttpResponseRedirect
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.contrib.auth.decorators import login_required
from django.contrib.auth.models import User
from django.utils.translation import ugettext as _
from django import forms
from django.utils.safestring import mark_safe

Then I went through and deleted all of these imports from throughout the project, and replaced it with a single import:

from avatar import dj

Now we have that shortcut module loaded, we can use it. Here's an example of what one view looked like before:

from django.contrib.auth.decorators import login_required
from django.template import RequestContext
from django.shortcuts import render_to_response
from django.utils.translation import ugettext as _
from django.http import HttpResponseRedirect

def delete(request, extra_context={}, next_override=None):
    avatars = Avatar.objects.filter(user=request.user).order_by('-primary')
    if avatars.count() > 0:
        avatar = avatars[0]
    else:
        avatar = None
    delete_avatar_form = DeleteAvatarForm(request.POST or None, user=request.user)
    if request.method == 'POST':
        if delete_avatar_form.is_valid():
            ids = delete_avatar_form.cleaned_data['choices']
            Avatar.objects.filter(id__in=ids).delete()
            request.user.message_set.create(
                message=_("Successfully deleted the requested avatars."))
            return HttpResponseRedirect(next_override or _get_next(request))
    return render_to_response(
        'avatar/confirm_delete.html',
        extra_context,
        context_instance = RequestContext(
            request,
            { 'avatar': avatar,
              'avatars': avatars,
              'delete_avatar_form': delete_avatar_form,
              'next': next_override or _get_next(request), }
        )
    )
change = login_required(change)

And here's what the code looked like afterward:

from avatar import dj

def delete(request, extra_context={}, next_override=None):
    avatars = Avatar.objects.filter(user=request.user).order_by('-primary')
    if avatars.count() > 0:
        avatar = avatars[0]
    else:
        avatar = None
    delete_avatar_form = DeleteAvatarForm(request.POST or None, user=request.user)
    if request.method == 'POST':
        if delete_avatar_form.is_valid():
            ids = delete_avatar_form.cleaned_data['choices']
            Avatar.objects.filter(id__in=ids).delete()
            request.user.message_set.create(
                message=dj._("Successfully deleted the requested avatars."))
            return dj.HttpResponseRedirect(next_override or _get_next(request))
    return dj.render_to_response(
        'avatar/confirm_delete.html',
        extra_context,
        context_instance = dj.RequestContext(
            request,
            { 'avatar': avatar,
              'avatars': avatars,
              'delete_avatar_form': delete_avatar_form,
              'next': next_override or _get_next(request), }
        )
    )
change = dj.login_required(change)

It works! Although after all of that, I have decided that in my opinion it's just not worth the effort. It adds an extra level of indirection when tracing through the code, it litters the view with the dj namespace, and it's harder to know what you have available to you. Maybe you like this style better, though. If so, what about this style do you like?

Ahhh, Django: my favorite web framework. And CouchDB: my favorite new database technology. How can I pair these two awesomes together to make an awesome-er?

One of the features that I would like to add to this site when it's time for an upgrade is a lifestream. It seems like everyone is doing it these days (isn't this great logic!), so I probably should too. Originally this was going to be written in the standard Django way--write some models, fill it with data, and slice and dice that data to make it pretty.

After thinking about it, I decided not to go that route. Why? Well, let's go over it: There needs to be a Twitter model, that's for sure. I also want a Pownce model, and a Flickr model. Already this is becoming tedious! At this point we have two options: continue creating these individual models and fill them with data, or try to find the common bits and group them into Ubermodels of some sort, with some type of field to use as a discriminator. Ugh.

This is the perfect use case for a schemaless database, and CouchDB fits that bill just perfectly. Plus its python support is actually quite mature, and running it on a mac is, quite literally, one click. So now that we've all agreed (we all agree, right?) that we want to use CouchDB with Django, how can we make it happen?

First let's set some database settings:

COUCHDB_HOST = 'http://localhost:5984/'
TWITTER_USERNAME = 'ericflo'

So far, so good. Now let's write some initialization code and put it in to an application in the __init__.py:

from couchdb import client
from django.conf import settings

class CouchDBImproperlyConfigured(Exception):
    pass

try:
    HOST = settings.COUCHDB_HOST
except AttributeError:
    raise CouchDBImproperlyConfigured("Please ensure that COUCHDB_HOST is " + \
        "set in your settings file.")

DATABASE_NAME = getattr(settings, 'COUCHDB_DATABASE_NAME', 'couch_lifestream')
COUCHDB_DESIGN_DOCNAME = getattr(settings, 'COUCHDB_DESIGN_DOCNAME',
    'couch_lifestream-design')

if not hasattr(settings, 'couchdb_server'):
    server = client.Server(HOST)
    settings.couchdb_server = server

if not hasattr(settings, 'couchdb_db'):
    try:
        db = server.create(DATABASE_NAME)
    except client.ResourceConflict:
        db = server[DATABASE_NAME]
    settings.couchdb_db = db

In this code, we're loading the CouchDB client and either creating or connecting to a database. We do a bit of error checking to ensure that if we forgot to add COUCHDB_HOST in our settings file, it will yell at us. So how do we use this? Let's write some data importing stuff!

try:
    import simplejson as json
except ImportError:
    import json

TWITTER_USERNAME = getattr(settings, 'TWITTER_USERNAME', None)

fetched = urlopen('http://twitter.com/statuses/user_timeline.json?id=%s' % (
    TWITTER_USERNAME,)).read()
data = json.loads(fetched)
map_fun = 'function(doc) { emit(doc.id, null); }'
for item in data:
    item['item_type'] = 'twitter'
    if len(db.query(map_fun, key=item['id'])) == 0:
        db.create(item)

This can go inside a Django management command or in a standalone script. Essentially what we're doing is loading the timeline for a user, and then for each item in that response we're setting the item_type to 'twitter'. Then we're checking to see if an item with that current twitter id already exists, and if not, we're creating it.

Now we need a way to query this data. In CouchDB, the way to query for data is using views. Views are stored in the database, so they can be entered manually, but I much prefer to manage views programmatically. Thankfully, Python's CouchDB library and Django give us all we need to make this very, very easy:

from django.db.models import signals
from couch_lifestream import models, db, COUCHDB_DESIGN_DOCNAME
from couchdb.design import ViewDefinition
from textwrap import dedent

by_date = ViewDefinition(COUCHDB_DESIGN_DOCNAME, 'by_date',
    dedent("""
    function(doc) {
        emit(doc.couch_lifestream_date, null);
    }
"""))

def create_couchdb_views(app, created_models, verbosity, **kwargs):
    ViewDefinition.sync_many(db, [by_date])
signals.post_syncdb.connect(create_couchdb_views, sender=models)

Make sure that this is placed somewhere that will be loaded when Django's manage.py is called. In this case, I put it in the __init__.py file under management/. What we're doing is creating two views--one which is keyed by the item_type (we set this earlier to be 'twitter'), and another which is keyed simply by date. When we run python manage.py syncdb, these views will automatically be re-synced with the database. Using this method, we are able to manage these views quickly and easily, and distribute them in a reusable way.

Now let's create some Django views so that we can visualize this data:

from couch_lifestream import db, COUCHDB_DESIGN_DOCNAME
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import Http404
from couchdb import client

def item(request, id):
    try:
        obj = db[id]
    except client.ResourceNotFound:
        raise Http404
    context = {
        'item': obj,
    }
    return render_to_response(
        'couch_lifestream/item.html',
        context,
        context_instance=RequestContext(request)
    )

def items(request):
    item_type_viewname = '%s/by_date' % (COUCHDB_DESIGN_DOCNAME,)
    lifestream_items = db.view(item_type_viewname, descending=True)
    context = {
        'items': list(lifestream_items),
    }
    return render_to_response(
        'couch_lifestream/list.html',
        context,
        context_instance=RequestContext(request)
    )

The item view is fairly self-explanatory. We query the db for the object of the specified id, and if it doesn't exist, we throw a 404. If it does exist, we throw it into the context and let the template render the page. The items view is slightly more interesting. In this case, we're using that CouchDB view that we created to query the database by date, and passing that list into the context.

Obviously there's a ton more that we could cover, but these basic building blocks that I've demonstrated are enough to get you started. After this it's mostly all presentational work. I've open sourced all of the code that has been written so far for the upcoming lifestream portion of this site, even though right now it only supports Twitter and Pownce. I plan on continuing work on it to support all of the services that I use. You can track my progress at the project's page.

I'll make sure to blog about this again once the project is more mature, but for now it should be fun to play around with. Are you using CouchDB with Django? If yes, then how are you dealing with that interaction?

Prompted by this whole blog post-per-day thing, I've begun looking at the source code for this site again. It was my first real Django project in early 2006, and I was excited about all of the features that it gave me. The community was much smaller then, and although I suspect the guys from Lawrence had a good idea of where site logic should go, at that time there was not the widespread knowledge of best practices.

No knowledge of best practices, still learning Django, and being a relative web development newbie, means that the code ended up being quite bad. It was all under one application, named 'blog'. All of the content loaded into the sidebar was done via context processors (one processor called back to this get_digg_rss function that I blogged about, for example), and all the templates were housed under that application.

Yeah. It was that bad.

But over the years, Django evolved, and some code that this blog used were deprecated while new functionality was added. Over time as things broke, little by little things were fixed and broken out into their own apps. In short, the site got better. It was always organic improvement, though, and never a deliberate effort. Meaning that while it's better than it was, it's still not very good.

Browsing through the code gives me a bit of nostalgia, but overwhelmingly my urge is to just delete the entire thing and start over, picking from the best reusable apps and migrating data as needed. After some consideration, I don't think that urge is a good urge to act upon. While it would take me a few afternoons to fix the code of the site, it would take several days to start from scratch.

This is one of my biggest faults as a programmer. Instead of working with ugly code and molding it into better code, I always enjoy starting from a clean slate and building it up from there--so much so that I'll sacrifice productivity to do so. I'm going to attempt to face this fault by not giving in to the temptation to rebuild, and instead to work with what I've got. Who knows, maybe I'll even open-source it. I won't open-source it unless its quality is up to par with the other open source apps in the Django reusable app ecosystem.

How do you deal with legacy code? Do you have any tips or tricks on how to do it? Do you think that it's better to just start from scratch?

A few years back a non-technical person asked me to explain what, exactly, the operating system did. I first started out by speaking in broad generalities, saying that it takes care of the basic needs of the computer and that it helps one thing talk to another. But that answer wasn't good enough. This person wanted me to go into more detail. What happens when I'm in the calculator application and I type 1+1 and hit enter? What things need to happen?

That really tripped me up. It's hard to even know where to start when someone who's curious like that asks a relatively innocent question like that. So I decided to just start from the basics. "A processor has these things called registers", I said. "And they store a tiny little amount of information, which they can do things with, like add and subtract." That seemed to basically satisfy them, but not me.

I made sure to say that, "You can't just keep everything in registers, because there are only a few of those. So you have to keep some data stored in this place with more space for data but it's slower." To which they responded, "Oh, so is that why when I add more RAM my computer gets faster?" "No," I said, "this is all still on the processor." "OK, so what's RAM then and why is it supposed to help?" they asked. "Because we can't store everything on the processor itself, because there's not enough space for data, so we have to store it in this place with much more space for data, but which is slower."

At this point I realized how redundant this conversation was going to get. It's the same thing for the hard drive, and (if you're old-school) tape drive or (if you're new-school) the internet. At the same point that person seemed to get bored--that moment of curiosity had passed. Computers just do their thing and we really didn't have to do anything special for all of that data movement to take place.

That's what struck me. When you come down to it, computers are just a waterfall of different caches, with systems that determine what piece of data goes where and when. For the most part, in user space, we don't care about much of that either. When writing a web application or even a desktop application, we don't much care whether a bit is in the register, the L1 or L2 cache, RAM, or if it's being swapped to disk. We just care that whatever system is managing that data, it's doing the best it can to make our application fast.

But then another thing struck me. Most web developers DO have to worry about the cache. We do it every day. Whether you're using memcached or velocity or some other caching system, everyone's manually moving data from the database to a cache layer, and back again. Sometimes we do clever things to make sure this cached data is correct, but sometimes we do some braindead things. We certainly manage the cache keys ourselves and we come up with systems again and again to do the same thing.

Does this strike anyone else as inconsistent? For practically every cache layer down the chain, a system (whose algorithms are usually the result of years and years of work and testing) automatically determines how to best utilize that cache, yet we do not yet have a good system for doing that with databases. Why do you think that is? Is it truly one of the two hard problems of computer science? Is it impossible to do this automatically? I honestly don't have the answers, but I'm interested if you do.

You won't see this in the Django documentation, you won't see it mentioned on other blogs, and you certainly won't hear the core developers of Django boasting about it, but Django's ORM has a secret weapon: it supports SQL group_by and having clauses, and it has for quite some time.

It's not part of the public QuerySet API, but rather a part of the private Query API. But just because it's not part of the public API doesn't mean that it's not easy to use--it just means that it might change in the future. So it's really a "use at your own risk" type of deal now. If you're up for keeping on top of things so that you know what to change when the next version of Django comes out, then read on. First, let's start with some model definition:

class TumbleItem(models.Model):
    title = models.CharField(max_length=255)
    item_type = models.CharField(max_length=50)

    def __unicode__(self):
        return '%s: "%s"' % (self.item_type, self.title)

A simple tumblog item. Very simple, as in, not really useful at all. But that's OK since this is just a demonstration. To demonstrate, let's create some data:

>>> ti1 = TumbleItem.objects.create(title='Blog Post 1', item_type='blog')
>>> ti2 = TumbleItem.objects.create(title='Blog Post 2', item_type='blog')
>>> ti3 = TumbleItem.objects.create(title='Blog Post 3', item_type='blog')
>>> ti4 = TumbleItem.objects.create(title='Article Dugg 1', item_type='digg')
>>> ti5 = TumbleItem.objects.create(title='Article Dugg 2', item_type='digg')
>>> ti6 = TumbleItem.objects.create(title='Link Saved 1', item_type='link')

OK now that we've loaded some data, let's use the group_by functionality!

>>> qs = TumbleItem.objects.all()
>>> qs.query.group_by = ['item_type']
>>> item_types = [i.item_type for i in qs]
>>> item_types
[u'blog', u'digg', u'link']

There we go, it's quick, easy, and it seems to Just Work. But let's try to grab only the item_types which have more than one item:

>>> qs = TumbleItem.objects.all()
>>> qs.query.group_by = ['item_type']
>>> qs.query.having = ['COUNT(item_type) > 1']
>>> item_types = [i.item_type for i in qs]
>>> item_types
[u'blog', u'digg']

And now we've successfully used the group_by and having functionality in the Django ORM. I'm excited for some aggregation functionality to start being exposed as a public API, as I'm sure it will be more elegant than this solution, but at the same time this is a neat hidden secret in the Django ORM. Well now you have the knowledge, so you have the power, and it's up to you to use it wisely!

Wow, that's quite a title! You're already probably queuing up all of your counterpoints and your rebuttals. In fact it's not quite that serious, but a seriously worrying trend is emerging that I'd like to address.

SQLAlchemy started out with a bang. When it came out, there was really nothing else out there that even came close in terms of features, performance, and ease of use. SQLObject was the reigning champion of Python ORMs, and the Django ORM was around but much less powerful at that time. So now it's 3 years later. Is SQLAlchemy still the best? I think it is, and here's why:

1. It has no preconception of how your database should be laid out. If you haven't created your database yet, you can choose to use the ActiveRecord-style layout of one object mapping to one table. If that's not the case for your application, you can even map arbitrary select statements onto objects, if that's how much flexibility you want. So already, SQLAlchemy is an option for thousands of people for which the other ORMs would not be.
2. It's tuned for performance. Whether it's the extremely slick connection pooling feature, or whether it's the Unit of Work design pattern which will issue as few queries as possible. Whether it's the ability to specify lazily or eagerly loaded columns, or whether you note the identity map which keeps track of all of the objects that are in memory, SQLAlchemy has lots of performance-tuning abilities. In my mind, SQLAlchemy gives you all the tools that you need to create a blazing fast application.
3. It has a wonderful community which is building great tools on top of SQLAlchemy itself. A clear example of this is the set of plugins that are shipped with SQLAlchemy, by default. These include a simple declarative layer, an association proxy, and SqlSoup, which uses introspection to map tables on the fly. These are great examples of how the community is creating great libraries that sit on top of SQLAlchemy. They also serve as proof of the configurability of the ORM itself.
4. It supports your database. According to its own website, "SQLAlchemy includes dialects for SQLite, Postgres, MySQL, Oracle, MS-SQL, Firebird, MaxDB, MS Access, Sybase and Informix; IBM has also released a DB2 driver." This is a pretty massive list and goes a long way in proving the power of the ORM.

It comes down to this. SQLAlchemy is a well-designed, performant, widely supported ORM which treats SQL with the respect it deserves. I haven't provided any code samples here because you should do that yourselves, with your own apps. It's just that good. So what are you waiting for?