zmq: consider loosening version constraints

[oliver-sanders]

Pyzmq/ZMQ is a dependency of Cylc. Cylc is a distributed system.

We currently pin the version of pyzmq (and consequently ZMQ?) to a major release in order to ensure that all Cylc components in a distributed installation are installed with the same version of ZMQ. We do this because we haven't investigated the compatibility policy for ZMQ so are on the side of caution.

Take a dive into the ZMQ docs to find out what their policy for forward/backward compatibility is to see what would be a sensible version range for Cylc going forward.

See also #5272

Pull requests welcome!

[oliver-sanders]

There's no direct statement saying pyzmq vx will work with vy, here's the chain of statements which leads to the implicit conclusion of interoperability...

---

Pyzmq supports v4 and later v3 versions of libzmq:

PyZMQ aims to support all stable (≥ 3.2.2, ≥ 4.0.1 ) versions of libzmq

-- pyzmq (retrieved 02-02-2023)

Libzmq implements the ZMTP protocol:

ZeroMQ core engine in C++, implements ZMTP/3.1

libzmq v4.3.4 (retrieved 02-02-2023)

The ZMTP protocol has built-in version detection with defined behaviour:

When we have detected a ZMTP 2.0 peer ... sending our version number... as defined by the ZMTP 2.0 specification.

When we have detected a ZMTP 3.0 peer ... and then continue with the security handshake as normal.

-- 23/ZMTP (retrieved 02-02-2023)

The ZMTP protocol at V3 leaves room for forward compatibility:

A peer MUST accept higher protocol versions as valid. That is, a ZMTP peer MUST accept protocol versions greater or equal to 3.1. This allows future implementations to safely interoperate with current implementations.

A peer SHALL always use its own protocol (including framing) when talking to an equal or higher protocol peer.

A peer MAY downgrade its protocol to talk to a lower protocol peer.

If a peer cannot downgrade its protocol to match its peer, it MUST close the connection.

-- http://zmtp.org/page:read-the-docs (retrieved 02-02-2023)

There was historically a period where this was not true:

Ah, the pain, the pain. I’m not speaking of the time at age eight when I stepped on a plank with a 4-inch nail protruding from it. That was relatively OK. I’m speaking of 2010-2011 when we had multiple parallel releases of ZeroMQ, each with different incompatible APIs or wire protocols. It was an exercise in bad rules, pointlessly enforced, that still hurts us today. The rule was, “If you change the API or protocol, you SHALL create a new major version”. Give me the nail through the foot; that hurt less.

-- zmq-guide (retrieved 02-02-2023)

0MQ/3.2.2 stable and later releases are compatible with 2.2 and 2.1. All other 3.x versions up to 3.2.1 are only compatible with themselves, which is unfortunate but caused by releases 3.0 and 3.1 which broke the protocol without adding any version information. One of the symptoms of using these incompatible releases (3.0.x, 3.1.x, 3.2.0, 3.2.1) with stable releases is that request-reply will work in one direction only.

-- zeromq (retrieved 02-02-2023)

However, that isn't relevant because:

PyZMQ aims to support all stable (≥ 3.2.2, ≥ 4.0.1 ) versions of libzmq

-- pyzmq (retrieved 02-02-2023)

---

Wow, that was a long chain.

So all this is a long-winded way of saying, yes, we should be good to loosen our version coupling.