Sat, 18 Mar 2006

A solution to RSS polling

The problem with RSS is that it requires polling. In order to see if a feed has changed, you have to download the feed and compare the timestamps to see if any entries have changed. This bothers a lot of people. First, because it's mostly wasted activity. Second because it increases the latency of a news feed. News isn't news once it's olds.

However, I think there is a very simple solution to RSS polling. When somebody connects to poll the RSS feed, don't give them the feed until it changes. The problem with this idea is that it results in every subscriber keeping an open TCP socket. Most operating systems dedicate (unswappable) operating system memory for this purpose. Most operating systems have trouble holding more than 10,000 connections open at one time.

However, those are trivial problems compared to the problem of reinventing a version of RSS that doesn't require polling. It is relatively trivial to change the software on a web server compared to the task of coordinating clients and servers to change together. So, for example, somebody who wanted to implement this idea could provision a special box whose only purpose was to serve up RSS feeds. It would poll a private RSS feed every few seconds or as configured. It would distribute this RSS feed to the public, holding open as many TCP connections as needed, using a SQL database to hold them if necessary.

Posted [11:00] [Filed in: opensource] [permalink] [Google for the title] [digg this]