erq - Lightweight message queues in erlang.


Introduction/Background
=======================

erq is an attempt to clone Kestrel in Erlang.  Kestrel is a
lightweight message queue written in Scala by Robey Pointer at
twitter.  It is a clone of starling which was written in Ruby by
Blaine Cooke at twitter previously.  One nifty feature of Starling
(and therefore Kestrel (and therefore erq)) is that it shares an
interface with memcache, so that any platform that has a memcache
client (i.e. most) automatically have a client for accessing any of
these platforms.

You can find kestrel here:
http://github.com/robey/kestrel/tree/master

And starling here:
http://github.com/starling/starling/tree/master

I first became aware of Kestrel when it was posted to reddit:
http://www.reddit.com/r/programming/comments/80ip6/kestrel_twitters_new_message_queue_written_in/

The reddit thread links to this post by Robey Pointer, the author of Kestrel:
http://robey.lag.net/2008/11/27/scarling-to-kestrel.html


Motivation
==========

Between the posts and the README in github for Kestrel there seemed to
be a divide between people that were impressed with the throughput
and/or power-to-weight ratio of the code and those that were
thoroughly unimpressed.  I don't know much about this type of
software, having never written it before but my gut told me that
better performance should be easy to achieve and that the lines of
code seemed very high.  This project is an attempt to confirm my
suspicions while getting more familiar with Erlang.


Status
======

So far I have implemented the basic protocol -- specifically just
"get" and "set".  I have a python test client that works against
Kestrel or erq (and presumably against starling as well, although I
haven't tested it).  Currently I have implemented persistence by
writing erlang terms to a file as text.  This is horribly inefficient
and has slowed things way down but it works.

As of 4/23/2009 I have added basically blocking gets, using the same
command format as Kestrel, specifically appending options (t for timeout
in this case) to the queue name separate by a slash, e.g.

	get foo/t=250

...this will block up to 250ms waiting for an item from the "foo" queue.

News
====

erq is in use on our production site at my day job.  I imagine we will
end up switching to something more mature and fully featured like ZeroMQ,
RabbitMQ, beanstalkd, or perhaps even Kestrel.  But, we needed a queueing
system to offload long-running AJAX request processing from our webserver
and erq was the easiest thing for me to get up and running quickly (since
I know it inside and out).  So far it's handling everything we need just
fine, so who knows, maybe it'll stick around.   This was already my motivation
for adding the blocking mode, so maybe it'll drive some new development
too...

Also I have updated the main module to export the serve function and call
itself in a fully qualified manner, so it is now possible to hot swap in
new code through the whole system.  This means you can turn debugging on
and off or deploy a new version without stopping the system.


Reference
=========

The memcache protocol:

http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt