Reminder: latest ≠ greatest

Reminder: latest ≠ greatest

This is not a new relevation; I have known this for a while now. Still, I made a careless mistake earlier in my Docker Compose YML file when copying from some examples:

  rabbitmq:
    image: rabbitmq:latest
    ...
  celery:
    ...
    depends_on:
      rabbitmq:
        condition: service_started

For the longest time, this worked fine. Then one day this week, my setup kept erroring out upon starting. The celery service failed to start up in a cycle of errors and restarts:

celery-1  | [2026-05-31 01:03:38,588: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672//
celery-1  | [2026-05-31 01:03:38,600: INFO/MainProcess] mingle: searching for neighbors
celery-1  | [2026-05-31 01:03:38,613: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
celery-1  | Traceback (most recent call last):
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 346, in start
celery-1  |     blueprint.start(self)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/bootsteps.py", line 116, in start
celery-1  |     step.start(parent)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/worker/consumer/mingle.py", line 37, in start
celery-1  |     self.sync(c)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/worker/consumer/mingle.py", line 41, in sync
celery-1  |     replies = self.send_hello(c)
celery-1  |               ^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/worker/consumer/mingle.py", line 54, in send_hello
celery-1  |     replies = inspect.hello(c.hostname, our_revoked._data) or {}
celery-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/app/control.py", line 389, in hello
celery-1  |     return self._request('hello', from_node=from_node, revoked=revoked)
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/app/control.py", line 106, in _request
celery-1  |     return self._prepare(self.app.control.broadcast(
celery-1  |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/celery/app/control.py", line 785, in broadcast
celery-1  |     return self.mailbox(conn)._broadcast(
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/kombu/pidbox.py", line 347, in _broadcast
celery-1  |     self._publish(command, arguments, destination=destination,
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/kombu/pidbox.py", line 309, in _publish
celery-1  |     maybe_declare(self.reply_queue(chan))
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/kombu/common.py", line 113, in maybe_declare
celery-1  |     return _maybe_declare(entity, channel)
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/kombu/common.py", line 155, in _maybe_declare
celery-1  |     entity.declare(channel=channel)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/kombu/entity.py", line 617, in declare
celery-1  |     self._create_queue(nowait=nowait, channel=channel)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/kombu/entity.py", line 626, in _create_queue
celery-1  |     self.queue_declare(nowait=nowait, passive=False, channel=channel)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/kombu/entity.py", line 655, in queue_declare
celery-1  |     ret = channel.queue_declare(
celery-1  |           ^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/channel.py", line 1162, in queue_declare
celery-1  |     return queue_declare_ok_t(*self.wait(
celery-1  |                                ^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/abstract_channel.py", line 99, in wait
celery-1  |     self.connection.drain_events(timeout=timeout)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/connection.py", line 526, in drain_events
celery-1  |     while not self.blocking_read(timeout):
celery-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/connection.py", line 532, in blocking_read
celery-1  |     return self.on_inbound_frame(frame)
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/method_framing.py", line 53, in on_frame
celery-1  |     callback(channel, method_sig, buf, None)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/connection.py", line 538, in on_inbound_method
celery-1  |     return self.channels[channel_id].dispatch_method(
celery-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/abstract_channel.py", line 156, in dispatch_method
celery-1  |     listener(*args)
celery-1  |   File "/code/.venv/lib/python3.12/site-packages/amqp/connection.py", line 668, in _on_close
celery-1  |     raise error_for_code(reply_code, reply_text,
celery-1  | amqp.exceptions.InternalError: Queue.declare: (541) INTERNAL_ERROR - Feature `transient_nonexcl_queues` is deprecated.
celery-1  | By default, this feature is not permitted anymore.
celery-1  | The feature will be removed from a future major RabbitMQ version, regardless of the configuration; actual version to be determined.
celery-1  | To...
celery-1  | [2026-05-31 01:03:38,638: WARNING/MainProcess] /code/.venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:397: CPendingDeprecationWarning: 
celery-1  | In Celery 5.1 we introduced an optional breaking change which
celery-1  | on connection loss cancels all currently executed tasks with late acknowledgement enabled.
celery-1  | These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
celery-1  | back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
celery-1  | setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.
celery-1  | 
celery-1  |   warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
celery-1  |

Since worker_cancel_long_running_tasks_on_connection_loss was the last thing in the logs, I looked into that first. And that was a terrific waste of time.

The actual problem is one earlier from RabbitMQ.

The Problem: RabbitMQ 4.2 → 4.3 Is a Breaking Change

RabbitMQ 4.3 (4.3.1 specifically) got picked up by Docker Compose this week (while it was 4.2.7 before). Despite a minor version bump (4.2 → 4.3) for RabbitMQ, this was a breaking change for Celery.

Here are the RabbitMQ logs when using RabbitMQ 4.2.7:

rabbitmq-1  | 2026-05-31 00:53:46.042455+00:00 [warning] <0.541.0> Deprecated features: `transient_nonexcl_queues`: Feature `transient_nonexcl_queues` is deprecated.
rabbitmq-1  | 2026-05-31 00:53:46.042455+00:00 [warning] <0.541.0> By default, this feature can still be used for now.
rabbitmq-1  | 2026-05-31 00:53:46.042455+00:00 [warning] <0.541.0> Its use will not be permitted by default in a future minor RabbitMQ version and the feature will be removed from a future major RabbitMQ version; actual versions to be determined.
rabbitmq-1  | 2026-05-31 00:53:46.042455+00:00 [warning] <0.541.0> To continue using this feature when it is not permitted by default, set the following parameter in your configuration:
rabbitmq-1  | 2026-05-31 00:53:46.042455+00:00 [warning] <0.541.0>     "deprecated_features.permit.transient_nonexcl_queues = true"
rabbitmq-1  | 2026-05-31 00:53:46.042455+00:00 [warning] <0.541.0> To test RabbitMQ as if the feature was removed, set this in your configuration:
rabbitmq-1  | 2026-05-31 00:53:46.042455+00:00 [warning] <0.541.0>     "deprecated_features.permit.transient_nonexcl_queues = false"

Note the warning entries about this transient_nonexcl_queues feature.

And for RabbitMQ 4.3.1:

rabbitmq-1  | 2026-05-31 00:30:27.549763+00:00 [error] <0.552.0> Deprecated features: `transient_nonexcl_queues`: Feature `transient_nonexcl_queues` is deprecated.
rabbitmq-1  | 2026-05-31 00:30:27.549763+00:00 [error] <0.552.0> By default, this feature is not permitted anymore.
rabbitmq-1  | 2026-05-31 00:30:27.549763+00:00 [error] <0.552.0> The feature will be removed from a future major RabbitMQ version, regardless of the configuration; actual version to be determined.
rabbitmq-1  | 2026-05-31 00:30:27.549763+00:00 [error] <0.552.0> To continue using this feature when it is not permitted by default, set the following parameter in your configuration:
rabbitmq-1  | 2026-05-31 00:30:27.549763+00:00 [error] <0.552.0>     "deprecated_features.permit.transient_nonexcl_queues = true"
rabbitmq-1  | 2026-05-31 00:30:27.550878+00:00 [error] <0.535.0> Error on AMQP connection <0.535.0> (172.21.0.4:60592 -> 172.21.0.2:5672, vhost: '/', user: 'guest', state: running), channel 1:
rabbitmq-1  | 2026-05-31 00:30:27.550878+00:00 [error] <0.535.0>  operation queue.declare caused a connection exception internal_error: "Feature `transient_nonexcl_queues` is deprecated.\nBy default, this feature is not permitted anymore.\nThe feature will be removed from a future major RabbitMQ version, regardless of the configuration; actual version to be determined.\nTo continue using this feature when it is not permitted by default, set the following parameter in your configuration:\n    \"deprecated_features.permit.transient_nonexcl_queues = true\""
rabbitmq-1  | 2026-05-31 00:30:27.552856+00:00 [info] <0.535.0> closing AMQP connection (172.21.0.4:60592 -> 172.21.0.2:5672, vhost: '/', user: 'guest', duration: '24ms')

Differences:

  • the transient_nonexcl_queues issues are flagged as errors, and
  • the connection is closed.

Since the connection is closed from RabbitMQ, Celery tries to reconnect, only to then be disconnected again. And the loop starts.

Lesson(s) Learned

Avoid Using “latest” or Unbound Upper Version in Specifiers

Because my Docker Compose file used rabbitmq:latest for the image version, it picked up 4.3.1 as soon as it was uploaded to Docker. From my point of view, what worked one day broke the next without me changing anything.

So, this is another reminder that I should never use latest in version specifiers in Docker Compose YMLs. More generally, I can imagine a similar problem with requirements.txt, pom.xml, pyproject.yml, package.json, etc. etc. etc., when the version specifier is defined too loosely.

A lot of tools, for whatever reason, decided not to use explicit versions when they help you add a dependency. For example, when adding a dependency to react, the tooling will add something like "react": "^19.0.0" instead of "react": "19.0.0". When adding a dependency to djangorestframework, the tooling will add something like "djangorestframework>=3.17.1" instead of "djangorestframework==3.17.1".

There is more to discuss of the pros and cons of explicit versioning. This post is not the place, and now is not the time.

For now, just a note: avoid using latest as version specifier in docker-compose.yml to avoid these surprises.

Old Lesson: Don’t Trust SEMVER Versions

One of many instances of “Rules for ye but not for me” when working in programming. Your manager/team will bark at you not correctly bumping major versions for breaking changes, but when others do it, eh–what are you gonna do?

Tags: , , ,