As I work on Scale0, I think it would be a good idea to share lessons I learn. I’ll also probably try to detail some design decisions. In fact, a lot has changed from the initial documentation I wrote, but that’s for another blog post (and doc updates).

To start with, this is based off of ZeroMQ 2.1.7 and pyzmq 2.1.7.

Right now I’m building ZeroMQ using Python, though I expect I may rewrite it in the future in another language like Erlang. I’m curious to see if I can get better performance out of the finished product in another language. However, I was encouraged to first build it in Python, a language I know and find all the problems. Thanks Zed Shaw for even more advice.

So the first lesson I learned was to use ZMQStream. One of the things I liked pyzmq was it borrowed Tornado’s ioloop. As you might know if you read my blog, I’m a big fan of Tornado. However, turns out I didn’t quite get how the ioloop worked with ZeroMQ.

So I built the following broker and worker.

Broker
Worker

All these do is send heartbeats to and from each other. The broker needs to be started first. Watching the output, you’ll see a list slowly grow on both ends. These are the missed heartbeats.

I broke down and asked for help on the ZeroMQ dev list, and got some help in the form of an explanation and pull request from Min RK. Basically, what was happening was on each loops execution I was only grabbing the top request on that POLLIN. The others were just not being handled. This is because in between each poll cycle multiple requests can be collected. ZMQStream natively handles this, it’s on_recv() is called for all requests on each cycle.

Here’s the current implementation, which if you run it you’ll see no heartbeats are missed.

Broker
Worker

Note: The socket implementation in the above examples is very much a work in progress. I’m already planning on switching to a PUB socket on the brokers and having workers connect to it with SUB sockets, instead of keeping track of connection sockets for every server. This will help optimize heartbeats going out as well as reduce the amount of information the broker needs to store about each worker.

I’m having fun working with ZeroMQ. It definitely makes you think and is a pretty amazing tool.