(solved)netbox upgrade from 4.0.7 to 4.0.11 netbox-rq failed

[maxzapp]

Hi,

i have a vm with 4.0.7 (and production data) and i want to upgrade (option A) to 4.0.11.

os-release

PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu

Steps to reproduce : follow the official updating procedure and restart netbox and netbox-rq service at the end as mentioned by the upgrade.sh script

Result: netbox-rq service not active (BUT: login possible and data available)

Expected Result: both netbox and netbox-rq are active as with v4.0.7

some checks i made:

systemctl status netbox-rq

● netbox-rq.service - NetBox Request Queue Worker
Loaded: loaded (/etc/systemd/system/netbox-rq.service; enabled; preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2024-11-14 14:35:41 CET; 24s ago
Docs: https://docs.netbox.dev/
Process: 4387 ExecStart=/opt/netbox/venv/bin/python3 /opt/netbox/netbox/manage.py rqworker high default low (code=exited, status=1/FAILURE)
Main PID: 4387 (code=exited, status=1/FAILURE)
CPU: 2.417s

journalctl -eu netbox-rq
File "/opt/netbox/venv/lib/python3.12/site-packages/django_rq/management/commands/rqworker.py", line 5, in
Nov 14 14:36:14 netbox python3[4449]: from rq import Connection
Nov 14 14:36:14 netbox python3[4449]: ImportError: cannot import name 'Connection' from 'rq' (/opt/netbox/venv/lib/python3.12/site-packages/rq/init.py). Did you mean: 'connections'?
Nov 14 14:36:14 netbox systemd[1]: netbox-rq.service: Main process exited, code=exited, status=1/FAILURE
Nov 14 14:36:14 netbox systemd[1]: netbox-rq.service: Failed with result 'exit-code'.
Nov 14 14:36:14 netbox systemd[1]: netbox-rq.service: Consumed 2.461s CPU time.
Nov 14 14:36:44 netbox systemd[1]: netbox-rq.service: Scheduled restart job, restart counter is at 15.
Nov 14 14:36:44 netbox systemd[1]: Started netbox-rq.service - NetBox Request Queue Worker.
Nov 14 14:36:47 netbox python3[4510]: Traceback (most recent call last):
Nov 14 14:36:47 netbox python3[4510]: File "/opt/netbox/netbox/manage.py", line 10, in
Nov 14 14:36:47 netbox python3[4510]: execute_from_command_line(sys.argv)
Nov 14 14:36:47 netbox python3[4510]: File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/init.py", line 442, in execute_from_command_line
Nov 14 14:36:47 netbox python3[4510]: utility.execute()
Nov 14 14:36:47 netbox python3[4510]: File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/init.py", line 436, in execute
Nov 14 14:36:47 netbox python3[4510]: self.fetch_command(subcommand).run_from_argv(self.argv)
Nov 14 14:36:47 netbox python3[4510]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 14:36:47 netbox python3[4510]: File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/init.py", line 275, in fetch_command
Nov 14 14:36:47 netbox python3[4510]: klass = load_command_class(app_name, subcommand)
Nov 14 14:36:47 netbox python3[4510]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 14:36:47 netbox python3[4510]: File "/opt/netbox/venv/lib/python3.12/site-packages/django/core/management/init.py", line 48, in load_command_class
Nov 14 14:36:47 netbox python3[4510]: module = import_module("%s.management.commands.%s" % (app_name, name))
Nov 14 14:36:47 netbox python3[4510]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 14:36:47 netbox python3[4510]: File "/usr/lib/python3.12/importlib/init.py", line 90, in import_module
Nov 14 14:36:47 netbox python3[4510]: return _bootstrap._gcd_import(name[level:], package, level)
Nov 14 14:36:47 netbox python3[4510]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 14 14:36:47 netbox python3[4510]: File "", line 1387, in _gcd_import
Nov 14 14:36:47 netbox python3[4510]: File "", line 1360, in _find_and_load
Nov 14 14:36:47 netbox python3[4510]: File "", line 1331, in _find_and_load_unlocked
Nov 14 14:36:47 netbox python3[4510]: File "", line 935, in _load_unlocked
Nov 14 14:36:47 netbox python3[4510]: File "", line 995, in exec_module
Nov 14 14:36:47 netbox python3[4510]: File "", line 488, in _call_with_frames_removed
Nov 14 14:36:47 netbox python3[4510]: File "/opt/netbox-4.0.11/netbox/core/management/commands/rqworker.py", line 3, in
Nov 14 14:36:47 netbox python3[4510]: from django_rq.management.commands.rqworker import Command as _Command
Nov 14 14:36:47 netbox python3[4510]: File "/opt/netbox/venv/lib/python3.12/site-packages/django_rq/management/commands/rqworker.py", line 5, in
Nov 14 14:36:47 netbox python3[4510]: from rq import Connection
Nov 14 14:36:47 netbox python3[4510]: ImportError: cannot import name 'Connection' from 'rq' (/opt/netbox/venv/lib/python3.12/site-packages/rq/init.py). Did you mean: 'connections'?
Nov 14 14:36:47 netbox systemd[1]: netbox-rq.service: Main process exited, code=exited, status=1/FAILURE
Nov 14 14:36:47 netbox systemd[1]: netbox-rq.service: Failed with result 'exit-code'.
Nov 14 14:36:47 netbox systemd[1]: netbox-rq.service: Consumed 2.448s CPU time.

---

System Status after (!) upgrade

System Status
NetBox version 4.0.11
Python version 3.12.3
Django version 5.0.9
PostgreSQL version PostgreSQL 16.4
Database name netbox
Database size 38 MB
RQ workers 0 (default queue)
System time 2024-11-14 13:41:30 UTC

[blipnet]

Hi, I had the same issue, upgrade django-rq and it will work

[candlerb]

Please explain what you are proposing more clearly.

• What version of django-rq did you have before?
• What version of django-rq did you upgrade to, that solved the problem?
• What version of django-rq is defined in /opt/netbox/requirements.txt ?

I am on Netbox 4.1.6 and I have exactly the versions which are defined in the requirements:

# /opt/netbox/venv/bin/pip freeze | grep -i rq
django-rq==2.10.2
rq==1.16.2
# grep rq /opt/netbox/requirements.txt
django-rq==2.10.2
rq==1.16.2

requirements.txt defines the versions that ./upgrade.sh will install.

It is possible that some other package that django-rq depends on was affected; in that case, I'd want to see the full output of /opt/netbox/venv/bin/pip freeze from before and after.

@maxzapp: if you still have this problem, please can you run /opt/netbox/venv/bin/pip freeze and post the full output here.

Here is mine:

# /opt/netbox/venv/bin/pip freeze
asgiref==3.8.1
attrs==24.2.0
babel==2.16.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
click==8.1.7
colorama==0.4.6
cryptography==43.0.3
defusedxml==0.8.0rc2
Django==5.0.9
django-cors-headers==4.6.0
django-debug-toolbar==4.4.6
django-filter==24.3
django-graphiql-debug-toolbar==0.2.0
django-htmx==1.21.0
django-js-asset==2.2.0
django-mptt==0.16.0
django-pglocks==1.0.4
django-prometheus==2.3.1
django-redis==5.4.0
django-rich==1.13.0
django-rq==2.10.2
django-tables2==2.7.0
django-taggit==6.1.0
django-timezone-field==7.0
djangorestframework==3.15.2
drf-spectacular==0.27.2
drf-spectacular-sidecar==2024.7.1
feedparser==6.0.11
ghp-import==2.1.0
graphene==3.4.1
graphene-django==3.2.2
graphql-core==3.2.5
graphql-relay==3.2.0
gunicorn==23.0.0
idna==3.10
inflection==0.5.1
Jinja2==3.1.4
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
Markdown==3.7
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdurl==0.1.2
mergedeep==1.3.4
mkdocs==1.6.1
mkdocs-autorefs==1.2.0
mkdocs-get-deps==0.2.0
mkdocs-material==9.5.43
mkdocs-material-extensions==1.3.1
mkdocstrings==0.26.2
mkdocstrings-python-legacy==0.2.4
netaddr==1.3.0
netbox-plugin-prometheus-sd==1.1.0
netbox-topology-views==4.1.0
nh3==0.2.18
oauthlib==3.2.2
packaging==24.1
paginate==0.5.7
pathspec==0.12.1
pillow==11.0.0
platformdirs==4.3.6
prometheus_client==0.21.0
promise==2.3
psycopg==3.2.3
psycopg-c==3.2.3
psycopg-pool==3.2.3
pycparser==2.22
Pygments==2.18.0
PyJWT==2.9.0
pymdown-extensions==10.12
python-dateutil==2.9.0.post0
python3-openid==3.2.0
pytkdocs==0.16.2
PyYAML==6.0.2
pyyaml_env_tag==0.1
redis==5.2.0
referencing==0.35.1
regex==2024.9.11
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.9.4
rpds-py==0.20.1
rq==1.16.2
sgmllib3k==1.0.0
six==1.16.0
social-auth-app-django==5.4.2
social-auth-core==4.5.4
sqlparse==0.5.1
strawberry-graphql==0.247.0
strawberry-graphql-django==0.49.1
svgwrite==1.4.3
tablib==3.7.0
text-unidecode==1.3
typing_extensions==4.12.2
tzdata==2024.2
uritemplate==4.1.1
urllib3==2.2.3
watchdog==6.0.0
wheel==0.44.0

But note that I have netbox-plugin-prometheus-sd and netbox-topology-views in /opt/netbox/local_requirements.txt so there may be additional dependencies there that wouldn't be in a vanilla install.

[maxzapp]

Hi,

yes i still have the problem, as i dont know how to properly upgrade django-rq, sudo apt install django-rq ?

actual django-rq according the requirements.txt django-rq==2.10.2

Pip freeze list:
/opt/netbox$ /opt/netbox/venv/bin/pip freeze
asgiref==3.8.1
attrs==24.2.0
babel==2.16.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.4.0
click==8.1.7
colorama==0.4.6
cryptography==43.0.3
defusedxml==0.8.0rc2
Django==5.0.9
django-cors-headers==4.4.0
django-debug-toolbar==4.4.6
django-filter==24.2
django-graphiql-debug-toolbar==0.2.0
django-htmx==1.19.0
django-js-asset==2.2.0
django-mptt==0.16.0
django-pglocks==1.0.4
django-prometheus==2.3.1
django-redis==5.4.0
django-rich==1.11.0
django-rq==2.10.2
django-tables2==2.7.0
django-taggit==5.0.1
django-timezone-field==7.0
djangorestframework==3.15.2
drf-spectacular==0.27.2
drf-spectacular-sidecar==2024.7.1
feedparser==6.0.11
ghp-import==2.1.0
graphene==3.4.3
graphene-django==3.2.2
graphql-core==3.2.5
graphql-relay==3.2.0
gunicorn==23.0.0
idna==3.10
inflection==0.5.1
Jinja2==3.1.4
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
Markdown==3.7
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdurl==0.1.2
mergedeep==1.3.4
mkdocs==1.6.1
mkdocs-autorefs==1.2.0
mkdocs-get-deps==0.2.0
mkdocs-material==9.5.34
mkdocs-material-extensions==1.3.1
mkdocstrings==0.26.0
mkdocstrings-python-legacy==0.2.4
netaddr==1.3.0
nh3==0.2.18
oauthlib==3.2.2
packaging==24.2
paginate==0.5.7
pathspec==0.12.1
pillow==10.4.0
platformdirs==4.3.6
prometheus_client==0.21.0
promise==2.3
psycopg==3.2.1
psycopg-c==3.2.1
psycopg-pool==3.2.3
pycparser==2.22
Pygments==2.18.0
PyJWT==2.9.0
pymdown-extensions==10.12
python-dateutil==2.9.0.post0
python3-openid==3.2.0
pytkdocs==0.16.2
PyYAML==6.0.2
pyyaml_env_tag==0.1
redis==5.2.0
referencing==0.35.1
regex==2024.11.6
requests==2.32.3
requests-oauthlib==2.0.0
rich==13.9.4
rpds-py==0.21.0
rq==2.0.0
sgmllib3k==1.0.0
six==1.16.0
social-auth-app-django==5.4.2
social-auth-core==4.5.4
sqlparse==0.5.2
strawberry-graphql==0.239.2
strawberry-graphql-django==0.47.1
svgwrite==1.4.3
tablib==3.6.1
text-unidecode==1.3
typing_extensions==4.12.2
tzdata==2024.1
uritemplate==4.1.1
urllib3==2.2.3
watchdog==6.0.0
wheel==0.45.0

[candlerb]

Here are the differences: mine with -, yours with +

--- pipfreeze-candlerb	2024-11-15 13:08:56
+++ pipfreeze-maxzapp	2024-11-15 13:08:42
@@ -9,27 +9,27 @@
 cryptography==43.0.3
 defusedxml==0.8.0rc2
 Django==5.0.9
-django-cors-headers==4.6.0
+django-cors-headers==4.4.0
 django-debug-toolbar==4.4.6
-django-filter==24.3
+django-filter==24.2
 django-graphiql-debug-toolbar==0.2.0
-django-htmx==1.21.0
+django-htmx==1.19.0
 django-js-asset==2.2.0
 django-mptt==0.16.0
 django-pglocks==1.0.4
 django-prometheus==2.3.1
 django-redis==5.4.0
-django-rich==1.13.0
+django-rich==1.11.0
 django-rq==2.10.2
 django-tables2==2.7.0
-django-taggit==6.1.0
+django-taggit==5.0.1
 django-timezone-field==7.0
 djangorestframework==3.15.2
 drf-spectacular==0.27.2
 drf-spectacular-sidecar==2024.7.1
 feedparser==6.0.11
 ghp-import==2.1.0
-graphene==3.4.1
+graphene==3.4.3
 graphene-django==3.2.2
 graphql-core==3.2.5
 graphql-relay==3.2.0
@@ -47,24 +47,22 @@
 mkdocs==1.6.1
 mkdocs-autorefs==1.2.0
 mkdocs-get-deps==0.2.0
-mkdocs-material==9.5.43
+mkdocs-material==9.5.34
 mkdocs-material-extensions==1.3.1
-mkdocstrings==0.26.2
+mkdocstrings==0.26.0
 mkdocstrings-python-legacy==0.2.4
 netaddr==1.3.0
-netbox-plugin-prometheus-sd==1.1.0
-netbox-topology-views==4.1.0
 nh3==0.2.18
 oauthlib==3.2.2
-packaging==24.1
+packaging==24.2
 paginate==0.5.7
 pathspec==0.12.1
-pillow==11.0.0
+pillow==10.4.0
 platformdirs==4.3.6
 prometheus_client==0.21.0
 promise==2.3
-psycopg==3.2.3
-psycopg-c==3.2.3
+psycopg==3.2.1
+psycopg-c==3.2.1
 psycopg-pool==3.2.3
 pycparser==2.22
 Pygments==2.18.0
@@ -77,25 +75,25 @@
 pyyaml_env_tag==0.1
 redis==5.2.0
 referencing==0.35.1
-regex==2024.9.11
+regex==2024.11.6
 requests==2.32.3
 requests-oauthlib==2.0.0
 rich==13.9.4
-rpds-py==0.20.1
-rq==1.16.2
+rpds-py==0.21.0
+rq==2.0.0
 sgmllib3k==1.0.0
 six==1.16.0
 social-auth-app-django==5.4.2
 social-auth-core==4.5.4
-sqlparse==0.5.1
-strawberry-graphql==0.247.0
-strawberry-graphql-django==0.49.1
+sqlparse==0.5.2
+strawberry-graphql==0.239.2
+strawberry-graphql-django==0.47.1
 svgwrite==1.4.3
-tablib==3.7.0
+tablib==3.6.1
 text-unidecode==1.3
 typing_extensions==4.12.2
-tzdata==2024.2
+tzdata==2024.1
 uritemplate==4.1.1
 urllib3==2.2.3
 watchdog==6.0.0
-wheel==0.44.0
+wheel==0.45.0

You have some different versions to what I have, but mostly they match the requirements.txt from Netbox v4.0.11. In the cases where yours are newer (graphene, packaging, regex, rds-py, rq, sqlparse, sheel) it's because they're not pinned in requirements.txt in v4.0.11

Ah: I think I see it now. You have rq==2.0.0 instead of rq==1.16.2. This was a floating requirement, until Netbox v4.1.5 which added the explicit pinned version.

You can either pin this version yourself in local_requirements.txt, or you can upgrade to Netbox v4.1.6

Speaking personally: I've always advocated for Netbox builds to be fully reproducible, using pip freeze to ensure that everyone who installs Netbox gets exactly the same set of dependencies as anyone else, regardless of when or where they install it. This has been always been rejected by the maintainers.

But this causes problems time and again, especially people who are installing versions older than the latest bleeding edge (and I know v4.0.11 isn't really that old). It's just completely non-reproducible as-is.

I think I will decide to stop trying to support anyone who has a version mismatch now. The project has made its decision, and it needs to deal with the consequences.

EDIT: if you do want to install or upgrade to any version of Netbox other than the latest, there's a third-party repository of pip freezes which give known working combinations of packages. You can replace the supplied requirements.txt with these to get a reproducible build. https://github.com/markkuleinio/netbox-freezes

[maxzapp]

Hi ,

thanks for your support and i would regred if you stop to support others.
I was following the official netbox documentation so i dont know where these difference are coming from, the only thing i use are plugins from their official page.

If i try to upgrade to 4.1.6 the upgrade script aborts totally and my system is down.

thats why i stick with 4.0.x

[maxzapp]

Hi ,

ok this was the solution in this case :
add rq==1.16.2 in the local_requirements.txt and run sudo ./upgrade.sh
After sudo systemctl restart netbox netbox-rq both are active.

Many thanks to @candlerb and btw i learned again something new