Error upgrading from 4.0.11 to 4.1.0 #17529
Replies: 32 comments 9 replies
-
Bug reports are not to be used for upgrade support. Per the bug report template:
We know that simply installing NetBox v4.0.11 and upgrading to v4.1.0 works as expected, so you'll need to identify steps that someone else can take to reproduce this issue if you want it to be investigated as a bug. These steps must not involve the Docker container, as that is out of scope for this project. Once you've done that, please update the "Steps to Reproduce" section in your post above with the necessary detail. |
Beta Was this translation helpful? Give feedback.
-
Very strange approach, migration procedures are kind of part of the product, aren't they?
I have provided several links to issues and discussions where other users have encountered the same problem.
As I wrote above, issues in the netbox-docker repository are closed without review and indicate the need to create an issue in that repository.
I and other users encountering the problem have no other steps to take other than trying to upgrade the product.
|
Beta Was this translation helpful? Give feedback.
-
I never suggested otherwise. If you can point to a bug in the migration procedure, we'll happy to address it. But to do that, you first need to provide steps to reproduce the reported behavior, as I asked. Migrations can fail for any number of reasons, such as corrupt or missing data. This does not necessarily indicate a bug in NetBox. |
Beta Was this translation helpful? Give feedback.
-
I'd love to describe the specific steps, but there is only one step - trying to upgrade an existing netbox installation. Judging by the discussions, different users have tried different ways to reproduce the problem, but the result is the same.
That's true, but without help from developers, we're unlikely to find the problem. What is required of us to find the problem? And I want to pay attention to this point - judging by postgresql logs, during the migration procedure (0117_move_objectchange), first the table is renamed, and then an attempt to select by the old name is made: This seems a bit odd, would this not help in finding the problem? |
Beta Was this translation helpful? Give feedback.
-
|
Similar problem here, in-place upgrade didn't work, now I can't revert. |
Beta Was this translation helpful? Give feedback.
-
Ok, let's do that "one step." You said:
So let's do that:
All the migrations complete successfully, as expected: So, what's different about what you're doing? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for NetBox and the described steps. Since I have the same problem importing my NetBox 4.0.7 database (PostgreSQL 16.3) into NetBox 4.1.0 I have tested it with the v4.0 demo data and the same error appears as with my exported 4.0.7 database. Installation of NetBox 4.1.0 on a newly installed Ubuntu 24.04.1 LTS. NetBox runs as expected Loading the v4.0 demo data as described here or here into NetBox 4.1.0 then fix the permission problem with Now when I run |
Beta Was this translation helpful? Give feedback.
-
|
See #17379 (comment) From that, it appears to be that migration dependencies haven't been declared properly, so they are applied in a non-deterministic order. Hence it works for some people and not for others. Making this something that could be reproduced would be difficult: I don't know what affects the order in which they're applied (e.g. the order in which the files were created in the directory?) It might be a good idea if the migrations were sorted by name, so that any which don't explicitly depend on each other are still applied in a deterministic order, but that could involve changing the internals of the migrations library. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure about this. i.e. all code up to is executed normally.We can see it in postgre log (ALTER TABLE, ALTER INDEX... etc). But, when it comes to the update_content_types function a SELECT from an already deleted table happens (SELECT "extras_objectchange"."id",...).
But, I understand absolutely nothing about django and very little about python =( |
Beta Was this translation helpful? Give feedback.
-
|
Great suggestion about 0011_move_objectchange - #17388 (comment) After that, the netbox successfully updated and started up! |
Beta Was this translation helpful? Give feedback.
-
|
As I said in the comment linked by @avbor the problem is that extras.0117_move_objectchange depends on core.0011_move_objectchange running in the same migration run. This would not be a problem, but core.0011_move_objectchange
If the second migration fails, the user is left with a state that is not recoverable without manual intervention. I do not know if it is possible to prevent the failure on the second run. Reproduction:
|
Beta Was this translation helpful? Give feedback.
-
|
May or may not be related, but there is a reported dependency issue with This is on a database being upgraded from v2.11.11 to v3.7.8 schema. |
Beta Was this translation helpful? Give feedback.
-
|
After the problem has been recognized and a manual fix is ready. What would be the best course of action now? Better to wait if it is possible to fix this problem with a future update. Or make the appropriate fixes yourself with the risk that future updates could cause problems again. |
Beta Was this translation helpful? Give feedback.
-
We still have not identified the source of the failure. As demonstrated above, upgrading from v4.0.11 to v4.1.0 does not reproduce the issue; at least not with Python 3.10 and PostgreSQL 14. @amhn indicated in another thread that the initial failure was due to missing permissions:
It's not clear to me why this would be an issue if you're simply upgrading an existing database as suggested in the discussion above, as the necessary permissions should already be in place. As for backing out |
Beta Was this translation helpful? Give feedback.
-
I think it could be any of the problems in the migration process. For @amhn it was a permissions issue, for me I suspect it was a redis issue. |
Beta Was this translation helpful? Give feedback.
-
Just want to point out that I experienced that permissions issue with the public schema and never had it before. 4.0.10 was working perfectly for me, the install is bog standard, and no permissions should have changed prior to performing the upgrade. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I don't understand what you did to solve the problem. I did
Flushed the redis cache
Nothing worked for me. |
Beta Was this translation helpful? Give feedback.
-
|
Here is what I did to get this working again: The redis changes are probably only needed if you're using docker. The switch from redis to valkey causes the database to be different. It might be good to check the status of redis connectivity before the migration as a way to defend against this. Or something can watch for errors like these: |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @rfdrake that did the trick for me also! |
Beta Was this translation helpful? Give feedback.
-
|
I had the same issue in our staging environment, and rfdrake's fix worked. I was prepared to do it also in production, but the upgrade did go through in production without an issue. Maybe because I tried the upgrade before in staging with 4.1.4 or so. Now I used 4.1.6, because I had no time to investigate it when 4.1.4 came out. Both netbox instances running in kubernetes with a postgres 15.7 cluster. |
Beta Was this translation helpful? Give feedback.
-
|
Hello Robert, |
Beta Was this translation helpful? Give feedback.
-
|
I have the same problem. Upgrade from 4.0.11 to 4.1.0 using the steps from the Netbox documentation, Below the error from the upgrade. Running migrations:
Applying core.0011_move_objectchange... OK
Applying extras.0117_move_objectchange... OK
Applying extras.0118_customfield_uniqueness... OK
Applying extras.0119_notifications... OK
Applying circuits.0044_circuit_groups... OK
Applying core.0012_job_object_type_optional... OK
Applying extras.0120_eventrule_event_types... OK
Applying extras.0121_customfield_related_object_filter... OK
Applying dcim.0188_racktype... OK
Applying dcim.0189_moduletype_rack_airflow... OK
Applying dcim.0190_nested_modules... OK
Applying ipam.0070_vlangroup_vlan_id_ranges... OK
Applying virtualization.0039_virtualmachine_serial_number... OK
Applying virtualization.0040_convert_disk_size...Traceback (most recent call last):
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
return self.cursor.execute(sql, params)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/psycopg/cursor.py", line 97, in execute
raise ex.with_traceback(None)
psycopg.errors.NumericValueOutOfRange: integer out of range
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/netbox-4.1.0/netbox/manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
utility.execute()
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
self.execute(*args, **cmd_options)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
output = self.handle(*args, **options)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/base.py", line 107, in wrapper
res = handle_func(*args, **kwargs)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 356, in handle
post_migrate_state = executor.migrate(
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 135, in migrate
state = self._migrate_all_forwards(
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
state = self.apply_migration(
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
state = migration.apply(state, schema_editor)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 132, in apply
operation.database_forwards(
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
self.code(from_state.apps, schema_editor)
File "/opt/netbox-4.1.0/netbox/virtualization/migrations/0040_convert_disk_size.py", line 7, in convert_disk_size
VirtualMachine.objects.filter(disk__isnull=False).update(disk=F('disk') * 1000)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/models/query.py", line 1253, in update
rows = query.get_compiler(self.db).execute_sql(CURSOR)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1990, in execute_sql
cursor = super().execute_sql(result_type)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
cursor.execute(sql, params)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in execute
return self._execute_with_wrappers(
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 100, in _execute
with self.db.wrap_database_errors:
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 105, in _execute
return self.cursor.execute(sql, params)
File "/opt/netbox-4.1.0/venv/lib/python3.10/site-packages/psycopg/cursor.py", line 97, in execute
raise ex.with_traceback(None)
django.db.utils.DataError: integer out of rangeAnd from the postgres log. 2024-09-12 16:34:03.219 UTC [3939] netbox@netbox ERROR: integer out of range
2024-09-12 16:34:03.219 UTC [3939] netbox@netbox STATEMENT: UPDATE "virtualization_virtualmachine" SET "disk" = ("virtualization_virtualmachine"."disk" * 1000) WHERE "virtualization_virtualmachine"."disk" IS NOT NULLOnly difference is the Python version used. I have python 3.10 |
Beta Was this translation helpful? Give feedback.
-
|
@rfdrake Your steps worked perfectly to fix my netbox-docker upgrade from 4.0.11 -> 4.1.0. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I was finally able to successfully upgrade from 4.0.11 to 4.1.1 by fixing the broken permissions prior to running upgrade.sh. |
Beta Was this translation helpful? Give feedback.
-
It worked! Very many thanks! 🥇 |
Beta Was this translation helpful? Give feedback.
-
|
@KaleBrink had the same issue, |
Beta Was this translation helpful? Give feedback.
-
|
It's evident that this issue has become a dumping ground for various unrelated problems, so I'm going to convert it to a discussion. |
Beta Was this translation helpful? Give feedback.
-
|
Update: My issue solved. It is AWS ECS deployment. I use netbox-docker and change psycopg2-binary to psycopg[binary]. For anyone who runs into similiar migration issue, please check if correct psycopg is used in the environment. Ran into issue during migration |
Beta Was this translation helpful? Give feedback.
-
|
Docker deployment here. # stop all processes
docker-compose down
# list redis volumes
docker volume ls | grep redis
# remove the redis volumes listed above.
docker volume rm beta_netbox-redis-cache-data beta_netbox-redis-data
....
deploy netbox v4.1.x |
Beta Was this translation helpful? Give feedback.
-
|
This worked for me if I hadn't tried the upgrade yet. If I did and got the error, I needed to add these steps too. I went from netbox 4.0.8 to 4.1.4
|
Beta Was this translation helpful? Give feedback.
-
|
I ran into the same issue. @avbor 's tips helped me greatly. I did not run any permissions updates to fix the issue. I only ran essentially the following commands from @rfdrake - this was after I attempted to upgrade already. However, understanding that the "0011_move_objectchange" records only exist because of the failed migration, I suspect I ran into this with docker due to the redis -> valkey change. For anyone else stumbling upon this - try purging the redis volumes first before the upgrade. I am curious if that one simple step would avoid a lot of the headaches. |
Beta Was this translation helpful? Give feedback.
-
|
So I can verify if you perform the rm on the Redis volumes before you do the docker compose up, you won't have to do the manual DELETE of the migration and it will successfully complete. |
Beta Was this translation helpful? Give feedback.
-
|
I can also verify it this way: Just delete the two redis/redis-cache docker volumes before the first upgrade attempt and the upgrade works as expected. It will be successful and no other workarounds (deleting database table, setting permissions, etc.) needed. But only, if it's the really first upgrade attempt and no failed attempt was before! Unfortunately most folks will only recognize this after running into the issue. Adding a notice to the release notes for 3.0.2 would be great. Thanks :) |
Beta Was this translation helpful? Give feedback.
-
|
Hi, So here am i , stuck on 4.0.11as i don't understand the workarounds above. Output of last lines: |
Beta Was this translation helpful? Give feedback.
-
|
Don't know exactly how the non-docker installation looks like, but the workarounds effectively mean, that the cached data of the both REDIS instances have to be cleared before the very first update attempt. |
Beta Was this translation helpful? Give feedback.
-
|
I am not familiar with redis i just fond on google this redis-cli flushall I'll try It tomorrow |
Beta Was this translation helpful? Give feedback.
-
|
Had the same problem as you on a plain ubuntu non-docker installation. What then worked for me was fixing the broken permissions prior to running upgrade.sh. see here Link |
Beta Was this translation helpful? Give feedback.
-
|
@quantum909 that did not helped copg-pool-3.2.3 pycparser-2.22 pygments-2.18.0 pymdown-extensions-10.12 python-dateutil-2.9.0.post0 python3-openid-3.2.0 pytkdocs-0.16.2 pyyaml-env-tag-0.1 redis-5.2.0 referencing-0.35.1 regex-2024.11.6 requests-2.32.3 requests-oauthlib-2.0.0 rich-13.9.4 rpds-py-0.21.0 rq-1.16.2 sgmllib3k-1.0.0 six-1.16.0 social-auth-app-django-5.4.2 social-auth-core-4.5.4 sqlparse-0.5.1 strawberry-graphql-0.247.0 strawberry-graphql-django-0.49.1 svgwrite-1.4.3 tablib-3.7.0 text-unidecode-1.3 typing-extensions-4.12.2 tzdata-2024.2 uritemplate-4.1.1 urllib3-2.2.3 watchdog-6.0.0 The above exception was the direct cause of the following exception: Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
What worked for me was to not start netbox and redis containers but restore the database from 4.0.11 backup first, after that start all the containers. Netbox starts and runs migrations with no issues versions: |
Beta Was this translation helpful? Give feedback.
and others
-
Deployment Type
Self-hosted
NetBox Version
v4.1.0
Python Version
3.12
Steps to Reproduce
Trying to upgrade from 4.0.11 to 4.1.0 rise migration error.
Expected Behavior
No errors during upgrade.
Observed Behavior
Please don't close this issue without investigate, we have several closed unresolved issues in this repo and netbox-docker.
Like #17352 and netbox-community/netbox-docker#1314
In netbox-community/netbox-docker#1313 @tobiasge wrote
Needs to be fixed in Netbox itself.netbox-community/netbox-docker#1313 (comment)Also some discussions abount this problem - #17382 and #17388
Log from netbox side:
log from postgre:
Also, is it normal to first go renaming the table (
ALTER TABLE "extras_objectchange" RENAME TO "core_objectchange") and then select from the old name(SELECT "extras_objectchange"."id", "extras_objectchange"."time", "extras_objectchange"."user_name", "extras_objectchange"."request_id", "extras_objectchange"."action", "extras_objectchange"."changed_object_id", "extras_objectchange"."related_object_id", "extras_objectchange"."object_repr", "extras_objectchange"."prechange_data", "extras_objectchange"."postchange_data", "extras_objectchange"."changed_object_type_id", "extras_objectchange"."related_object_type_id", "extras_objectchange"."user_id" FROM "extras_objectchange" WHERE "extras_objectchange"."changed_object_type_id" IN (166) ORDER BY "extras_objectchange"."time" DESC)?Beta Was this translation helpful? Give feedback.
All reactions