A few months back GitHub rolled out its implementation of something that they call "Service Hooks". The idea behind these hooks is that when you commit some new piece of code to GitHub, they want to be able to alert other services that you have committed that code. For example, one of the service hooks is the ability to send a tweet to Twitter, and another of those hooks updates the Lighthouse ticket tracker.
I thought this was a really good idea when they rolled it out, so I did a bit of searching and found out that there is a larger body of work surrounding this idea, and that body of work is called Web Hooks. The central idea behind web hooks is that a user supplies a service that they use with a URL. Then, when that user performs an action on that service, the service agrees to send an HTTP POST request to the user's specified URL, with some information about the action that the user took on the service.
SlideShare has an excellent presentation deck about this idea, which likens it to Unix pipes. That analogy makes a lot of sense if you think about it. With the standard model that most websites follow today, a client can only send requests. This means repeated polling until the client receives the information that it is interested in. With web hooks, however, the service is responsible for passing that action along to the next service. This simple yet powerful mechanism can allow for very advanced systems which interoperate very simply through chaining.
Let's expore a concrete example of what this might look like. A few months back I signed up for a pro account on Flickr, so that I could upload some of the pictures that I had stored on my computer. What I did was to upload some pictures with descriptions, and then I went and posted on Twitter some of the links to those pictures. I also went and added that new Flickr account to FriendFeed so that others could see my pictures as well.
This was all a manual process. If both Flickr and Twitter supported web hooks, I could have simply set up their respective URLs and uploaded my pictures. The process might have happened like this: First, the pictures are uploaded. Then Flickr sends a POST request to Twitter, with the description of the picture and a link to the picture. Twitter sends a POST request to FriendFeed, adding the new item to my FriendFeed lifestream.
You could even write custom scripts to handle the web hooks. For example let's say that I want any tweet with the name 'Kevin' to be sent to my brother's email address. I could add a URL to Twitter linking to a script on my computer which scans the contents of the tweet. If the tweet has the name 'Kevin' in it, it would send an email. If not, it might do nothing.
I think that this concept is very powerful not only in terms of rendering trivial the interoperability between disparate services, but also in terms of simply saving on bandwidth and computing power. Technologies which constantly poll resources hoping for updated content seem silly in comparison to the powerful simplicity that web hooks provide.
There are definitely some drawbacks to a system like this. Firstly, the name: I actually can't think of a worse name for this concept. Web hooks?! Let's come up with something better. All joking aside though, this type of system does face a serious problem when it comes to the question of reliability. If a script receives no POST, it could mean that either no event happened, or that the internet connection went down for a bit, or that the service is down, or any number of other possible things. I think the solution for this is a hybrid model of sparse polling in conjunction with web hooks.
Most of all, this technology just seems so underused. There are ridiculously few people who implement something like this, yet it seems like an undeniably useful service--especially given its relative simplicity to implement. Let's all try to encourage the services that we use on a daily basis to support web hooks, because by doing just that, we can make the web a lot better.
All Content
By Alexander Artemenko at 1:23 a.m. on Nov. 27, 2008
I see another problem with Web Hooks. How do they deal with authentication?
In you example, flickr must to send POST to the twitter, but actually, to add a tweet, flickr must to authenticate himself with your login and password or to use some sort of the key based authentication.
In my opinion, this is the main obstacle, which does not allow to use WebHooks nowadays.
By Eric Florenzano at 1:31 a.m. on Nov. 27, 2008
I really don't think that it's a huge obstacle. Yes, it does make things slightly more complicated but if someone really wants to do authentication they can use some sort of token-based authentication mechanism like Flickr's API.
Actually, this would be perfect, because you could generate different auth tokens for each web hook and then have really fine-grained control over who has access to what, and you could change that access easily as well.
Again, yes it makes it a bit more complicated but I don't think it's a showstopper.
By Donovan Preston at 11:39 a.m. on Nov. 29, 2008
The callback URL should be a capability. If a client knows the URL (which should be unguessable) then the client is authorized to access the resource.
This can be accomplished with a capability proxy which simply maps uuids to back-end resources, or dynamic URLs which contain an encrypted segment signed with a private key known only to the server. Both of these approaches can also close over any state required, effectively making them continuations. One approach stores the state in the server, the other in the client.
By Dave Jeffery at 2 a.m. on Nov. 27, 2008
It would make a lot of sense for somebody like FriendFeed to implement Web Hooks. Perhaps it's a bit counter-intuitive since FF still have to poll the twitter/flickr/digg/whatever servers but if the service becomes popular then the twitters of this world will sit up and take notice (although hopefully twitter will take care of their crappy API auth system first). The FF solution would at least put an end to the constant polling that us guys who have lifestreams do.
And I don't mind 'Web Hooks', it's easy to understand and kinda catchy. It's got to be better than another TLA (Three Letter Acronym).
By Yashh at 2:12 a.m. on Nov. 27, 2008
Certainly a cool technique, but I can see right away the hardship in implementing this.
By Lawrence Oluyede at 8:32 a.m. on Nov. 27, 2008
That's a fine idea but it's a no no in practice for security and scalable reasons. Think about what FriendFeed does already. It uses RSS/Atom feeds to hookup with external services, asynchronously. Cross posting (and maybe synchronously) throughout different websites does not seem a good idea to me.
By Leah Culver at 9:53 a.m. on Nov. 27, 2008
Join the google group!
http://groups.google.com/group/webhooks/
It's super low volume (as of now).
There are a couple of ideas floating around about how to facilitate two-way communication between web services. I agree that auth and scalability are issues, although they could be worked out with a little thought.
I think the key factor that makes web hooks interesting is that it's done over HTTP which is really accessible and easy (as oppose to say, XMPP).
I also love that PayPal's IPN is considered one of the best examples of web hooks.
By Dave K at 6:43 p.m. on Nov. 27, 2008
The client request driven nature of the web is a big security advantage. Part of the reason all these rich web apps are so popular is they work through restrictions like firewalls and proxies and don't require exposing things.
Otherwise we might as well all just run cross platform apps written in .net or java or python and let them run amok.
Perhaps you'd like a sockets system in javascript? That'd be delightfully bad. Putting a web server in a browser is adding a lot of complex code which could be exploited. You move risk from the servers managed by professionals to the clients managed by morons.
By Donovan Preston at 5:17 p.m. on Nov. 29, 2008
Scalability is not really an issue. They are normal http requests, scale them just like any other web service. Auth isn't an issue either if you use capabilities, as I mention above.
As for the idea of having raw socket Apis in javascript, it's already been done. But I think this misses the point -- this architecture is far more useful for servers talking to other servers. The paypal ipn example is a good real life widespread and easy to implement case. Since clients aren't always connected to the cloud it would be better to use a client agent server architecture, where the server was given a URL to the agent service, and the client drained events from the agent whenever it was connected.
By wholesale jewelry at 9:32 p.m. on May 5, 2009
It is really a cool technique,thks.
By wholesale fashion jewelry at 1:34 a.m. on May 25, 2009
It is really a cool technique。 thanks
By ben10 oyunları at 2:58 a.m. on May 25, 2009
this architecture is far more useful for servers talking to other servers. The paypal ipn example is a good real life widespread and easy to implement case. Since clients aren't always connected to the cloud it would be better to use a client agent server architecture,
By PC Tools Anti Spyware at 10:10 a.m. on May 26, 2009
This is one of the best.....nah, who am I kidding - THE best site on the net
By amany kile at 1:24 p.m. on June 8, 2009
As for the idea of having raw socket Apis in javascript, it's already been done. But I think this misses the point -- this architecture is far more useful for servers talking to other servers.
The paypal ipn example is a good real life widespread and easy to implement case. Since clients aren't always connected to the cloud it would be better to use a client agent server architecture, where the server was given a URL to the agent service, and the client drained events from the agent whenever it was connected.
By STORNEY YANG at 1:25 p.m. on June 8, 2009
As for the idea of having raw socket Apis in javascript, it's already been done.
But I think this misses the point -- this architecture is far more useful for servers talking to other servers.
The paypal ipn example is a good real life widespread and easy to implement case.
Since clients aren't always connected to the cloud it would be better to use a client agent server architecture, where the server was given a URL to the agent service, and the client drained events from the agent whenever it was connected now.
By WoW Power Leveling at 4:26 a.m. on June 13, 2009
A good topic, it is so interesting.
By plastic injection molding at 9:36 p.m. on June 13, 2009
nice blogging!
By how to make organic compost at 1:32 p.m. on June 15, 2009
Great stuff here! I love this site so informative!
By jordan shoes at 12:35 a.m. on June 20, 2009
i'd like to let you know that my favorite site is your blog now!
By Nike Shoes at 12:36 a.m. on June 20, 2009
good job!
By jordan shoes at 12:36 a.m. on June 20, 2009
good, thanks for your great post!
By ugg boots at 12:37 a.m. on June 20, 2009
you are really something!
By sexy costumes at 8:02 p.m. on June 22, 2009
I did notice that CPU usage is upto 100% constantly on both cores, even if the Django project is idle and not serving anything. It doesn't seem to affect my machine performance at all thought
By tiffany jewellery at 12:38 a.m. on June 25, 2009
Great stuff here! I love this site so informative!