2.22. 0MQ

0MQ is a middleware for versatile messaging communication. 0MQ implements enhanced sockets that provide an interface to communication patterns such as publish subscribe or pipeline. 0MQ exports a low level interface in C, intended for use by language binding implementations, and a multitude of high level interfaces in languages ranging from C and C++ to Haskell, intended for application use.

2.22.1. Sockets

0MQ sockets represent the communication endpoints. Each socket has a type, specified at the socket creation time, which describes the role of the socket in a particular communication pattern.

Socket Creation Functions

Low Level Functions. 

void *zmq_socket (void *context, int type);
int zmq_close (void *socket);

context

initialized library handle

type

role in communication pattern

High Level Functions. 

// Generic socket creation functions.
zsock_t *zsock_new (int type);
void zsock_destroy (zsock_t **self_p);

// Type specific socket creation functions.
zsock_t *zsock_new_pub (const char *endpoint);
zsock_t *zsock_new_sub (const char *endpoint, const char *subscribe);
zsock_t *zsock_new_req (const char *endpoint);
zsock_t *zsock_new_rep (const char *endpoint);

A socket can use multiple supported transports. Basic transport types are local communication within a process (inproc), communication over local pipe (ipc), communication over hypervisor transport (vmci), communication over TCP sockets (tcp), communication over IP multicast (pgm) and UDP multicast (epgm).

A socket is connected with a transport by calling zmq_bind for incoming connections and by calling zmq_connect for outgoing connections. A single socket can be connected multiple times to multiple transports.

Socket Connection Functions

Low Level Functions. 

int zmq_bind (void *socket, const char *endpoint);
int zmq_connect (void *socket, const char *endpoint);

inproc

communication within process

  • address is an arbitrary string

ipc

communication over local pipe

  • address is a local pipe file name

vmci

communication over hypervisor transport

  • address is a virtual machine identifier or hypervisor

  • address includes port number with function similar to TCP or UDP

tcp

TCP socket opened on demand

pgm

IP multicast to destination

epgm

UDP multicast to destination

High Level Functions. 

int zsock_bind (zsock_t *self, const char *format, ...);
int zsock_connect (zsock_t *self, const char *format, ...);

Actual transport operations are asynchronous. In particular, nodes can call zmq_bind and zmq_connect in any order, connections are established transparently including across network outages (except in process communication where binding must come before connecting).

Asynchronous transport operation is carried out by dedicated threads. The design expects the thread count to reflect the traffic volume, with one thread handling roughly a gigabyte of traffic per second, regardless of the connection count.

In contrast to the background transport operation, where a thread can handle multiple connections, each socket should be used by a single thread only. Multiple threads can communicate through the process local sockets.

Sockets are configured through socket options. These include low level connection options such as system buffer sizes, message size limits, high water marks (which determine when excess messages are discarded or threads blocked), transport reconnection intervals, thread affinity settings, message filters and more.

Socket Configuration Functions

Low Level Functions. 

int zmq_setsockopt (void *socket, int option_name, const void *option_value, size_t option_len);
int zmq_getsockopt (void *socket, int option_name, void *option_value, size_t *option_len);

ZMQ_SUBSCRIBE

subscription filter

ZMQ_SNDHWM, ZMQ_RCVHWM

high water mark

ZMQ_SNDBUF, ZMQ_RCVBUF

system buffer size

ZMQ_RECONNECT_IVL

transport reconnect interval

ZMQ_RECOVERY_IVL

multicast absence tolerance

ZMQ_AFFINITY

transport thread affinity

ZMQ_RATE

multicast data rate

High Level Functions. 

// Convert from high level to low level socket reference.
void *zsock_resolve (void *self);

Messages are byte arrays even in heterogeneous environments. Messages are delivered atomically, regardless of size. Multipart messages are supported. Regardless of transport, message delivery is not guaranteed.

Message Transport Functions

Low Level Functions. 

int zmq_msg_send (zmq_msg_t *msg, void *socket, int flags);
int zmq_msg_recv (zmq_msg_t *msg, void *socket, int flags);

  • messages are byte arrays

  • message delivery is atomic

  • multipart messages are supported

High Level Functions. 

zmsg_t *zmsg_recv (void *source);
int zmsg_send (zmsg_t **self_p, void *dest);

// Multipart message functions.
size_t zmsg_size (zmsg_t *self);
int zmsg_prepend (zmsg_t *self, zframe_t **frame_p);
int zmsg_append (zmsg_t *self, zframe_t **frame_p);
zframe_t *zmsg_pop (zmsg_t *self);
zframe_t *zmsg_first (zmsg_t *self);
zframe_t *zmsg_next (zmsg_t *self);

2.22.2. Patterns

0MQ patterns represent the communication topology. Each socket has a role in a patern.

The request reply pattern connects clients with the ZMQ_REQ socket type to servers with the ZMQ_REP socket type. The pattern expects alternating request and reply messages and provides round robin to multiple servers and fair queueing to multiple clients. An intermediary made from the ZMQ_ROUTER and ZMQ_DEALER socket types can be used in an extended variant of the pattern.

Synchronous Request Reply Pattern

Connects multiple clients to multiple servers.

ZMQ_REQ. 

  • used by client to send requests and receive replies

  • allows only alternating sequence of send and recv

  • round robin when multiple servers connected

  • blocks when no service available

ZMQ_REP. 

  • used by service to receive requests and send replies

  • allows only alternating sequence of recv and send

  • fair queueing among clients

Asynchronous Request Reply Pattern

Connects multiple clients to multiple servers through an intermediary.

ZMQ_ROUTER. 

  • receives requests from clients with fair queueing

  • prefixes requests with client identifier

  • delivers replies to identified client

ZMQ_DEALER. 

  • receives replies from servers with fair queueing

  • delivers requests to servers with round robin

The publish subscribe pattern connects publishers with the ZMQ_PUB socket type to subscribers with the ZMQ_SUB socket type. The pattern distributes messages from a publisher to all connected subscribers. An intermediary made from the ZMQ_XSUB and ZMQ_XPUB socket types can be used in an extended variant of the pattern.

Static Publish Subscribe Pattern

Connects multiple publishers to multiple subscribers.

ZMQ_PUB. 

  • delivers messages to connected subscribers

  • never blocks

ZMQ_SUB. 

  • receives messages from connected publishers

  • fair queueing among publishers

Dynamic Publish Subscribe Pattern

Connects multiple publishers to multiple subscribers through an intermediary.

ZMQ_XSUB. 

  • delivers subscription requests to connected publishers

  • receives messages from connected publishers

  • fair queueing among publishers

ZMQ_XPUB. 

  • receives subscription requests from connected subscribers

  • delivers messages to connected subscribers

  • fair queueing among subscribers

  • never blocks

The pipeline pattern connects task generators with the ZMQ_PUSH socket type to task processors with the ZMQ_PULL socket type. The pattern delivers messages from a task generator to a task processor.

Pipeline Pattern

Connects task generators to task processors.

ZMQ_PUSH. 

  • delivers messages to connected task processors

  • round robin among processors

ZMQ_PULL. 

  • receives messages from connected task generators

  • fair queueing among generators

The exclusive pair pattern connects two peers with the ZMQ_PAIR socket type. The pattern is designed for communication within process.

More draft socket patterns exist, such as client server (a thread safe alternative to request reply), radio dish (a thread safe alternative to publish subscribe), channel (a thread safe alternative to exclusive pair), or peer to peer.

2.22.3. References

  1. The 0MQ Guide. https://zguide.zeromq.org

  2. The 0MQ RFC Specifications. https://rfc.zeromq.org