NetBox v3.7 #130
Merged
NetBox v3.7 #130
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi, we have a beautiful 3.7.1 NB db ready to test with your new importer .. migration from NB to Nautobot is under discussions |
Hi @hchybz , would be great to be able to test it with your data. I can provide you with a few steps to run the import in your dev environment, if you are interested. Or would you like other way of cooperation? |
|
Hi Jan, ok to test your steps, with pleasure. Our dev env is hosted either on our laptops with docker or in a kube cluster. Would be simpler i think to test it with docker. |
Hi, here are instructions to import the NetBox data into the dev Nautobot database. First, you need to export the data from the NetBox database: ./manage.py dumpdata \
--traceback \
--format=json \
--indent=2 \
--natural-primary \
--natural-foreign \
--exclude=extras.ObjectChange \
--exclude=extras.Report \
--exclude=extras.Script \
--exclude=django_rq.Queue \
--output=/tmp/netbox_data.jsonThe most simple way to get Nautobot dev environment with NetBox Importer App running is to use included docker compose based dev environment: pip install invoke
git clone https://github.com/nautobot/nautobot-app-netbox-importer.git
cd nautobot-app-netbox-importer/
git checkout u/snaselj-napps-204-netbox-v3.7
cp development/creds.example.env development/creds.env
invoke buildAfter the build is done, you can start the dev environment with: invoke debug -s nautobot
# ... logs ....
nautobot-1 | Nautobot initialized!This runs Nautobot and services using docker compose and keeps the logs open. After the services are up and running, you should be able to access Nautobot at http://localhost:8080/ and login with the default credentials admin/admin. Now, keep the Nautobot running and open a new terminal window. Run the following commands to import the exported data dump into the Nautobot database: cd nautobot-app-netbox-importer/
# Necessary to copy the exported data to the nautobot-app-netbox-importer directory to access it from the container
cp <path-to-exported>/netbox_data.json ./netbox_data.json
invoke import-netbox \
--bypass-data-validation \
--print-summary \
--print-fields-mapping \
--save-json-summary-path=mappings.json \
--save-text-summary-path=mappings.txt \
--no-dry-run \
--file=netbox_data.jsonThe command above produces detailed debug log and mappings files. These mappings files doesn't contain any sensitive data and can be shared with us to help us improve the importer. If you have any questions or need help, feel free to ask. |
snaselj
commented
Feb 9, 2024
There was a problem hiding this comment.
@whitej6 This file shows diff in field mappings from 3.6 => 3.7. Who can help to identify how to map skipped fields?
NO IMPORTER => NO TARGET
There was a problem hiding this comment.
Here you can see the diff between 3.6 and 3.7:
|
Hi Jan, one of my devs, aka Helder, tested your importer yesterday. We arrived at a point where half-U devices are not recognized by the importer I never understood why NB team has implemented that feature since what we need in DC is not half-height devices but half-width. For one of our sites we have a an HPC cluster with half-width 100G Infiniband switches on top of racks.. My colleague implemented that in NB using the half height feature.
the device type
on out dev NB, we gona delete those half U devices to check wether the import process will finish or not Is there a way to implement correctly half width devices in nautobot ? /BR |
It's still a draft, will be cleaned up before marking ready for review. |
|
We found another issue, related to a "new" feature in NB : "Custom Field Choices".. my dev guy think it is the field : 'choice_set': 11 From what I can see in NB (and what I know about my OSK infra :-) ) .. "Openstack region" is clearly a string,
When I click on "osk_region Choices". ..it is choice #11 as seen in URL.. looks like if link was not followed.
|
| @@ -8199,9 +8813,26 @@ | |||
| } | |||
| } | |||
| }, | |||
| "extras.customfieldchoiceset": { | |||
There was a problem hiding this comment.
Is this different from Nautobot's extras.customfieldchoice model?
There was a problem hiding this comment.
Yes, it's different.
NetBox allows to define single CustomFieldChoiceSet with multiple choices, and assign it to multiple CustomFields.
Nautobot allows to define multiple CustomFieldChoice and each assign to the single CustomField.
Implemented here: #131
| @@ -8669,6 +9338,18 @@ | |||
| "default_value": null, | |||
| "disable_reason": "" | |||
| }, | |||
| "object_types": { | |||
There was a problem hiding this comment.
How does this differ from the content_types field?
There was a problem hiding this comment.
Seems to be the same, mapped
| "users.adminuser": { | ||
| "content_type": "users.adminuser", | ||
| "content_type_id": 103, | ||
| "users.netboxuser": { |
There was a problem hiding this comment.
Does this map to Nautobot's users.user?
There was a problem hiding this comment.
Seems like it's just a proxy model without any data in the DB.
| @@ -120,9 +120,18 @@ fcb61481-0da8-5412-b68b-2463016a017d P2-12B | ['Rack R204 (Row 2) and power pane | |||
| Total validation issues: 48 | |||
| - Content Types Mapping Deviations: ---------------------------------------------------------------- | |||
| Mapping deviations from source content type to Nautobot content type | |||
| account.usertoken => account.usertoken | Disabled with reason: Nautobot content type: `account.usertoken` not found | |||
There was a problem hiding this comment.
Is this different from Nautobot's users.token?
There was a problem hiding this comment.
Do we want to import user tokens from NetBox to Nautobot? I would rather skip importing passwords and tokens.
Hi @hchybz , thanks for reporting this. Importing |
Thanks for testing this. Nautobot currently supports only integer values for U heights. Will improve the importer to be able to import such devices, but there will be possibly some data loss. |
|
@snaselj to be able to continue testing, we will change/delete assets that are racks in "half height" positions. |
Hi @hchybz and @helderbetiol, I added some improvements, |
|
Hi @snaselj , Thank you for the improvements! We are now facing the following issue: Any ideas on how to fix it? |
pushed a quick fix, will clean it up yet but should fix this issue now. If anything else, let me know please. |
Hi @snaselj Thank you for the quick fix, this time it went a little further but it still encountered an issue soon after: |
Hi @helderbetiol , Thank you for reporting this issue. The error indicates that some record references It's possible that the I've implemented a temporary fix, though I am still not sure about the root cause. Let me know if the issue persists please. Anyway, would be helpful if you could check the import log for the UUID |
|
Hi @snaselj , I'm still getting an error even with the most recent change: In the import logs, I see the same UUID being used before: |
Hi @helderbetiol , Thanks for the report. Could you please provide more details from the import logs? There should be a line Could you please check, that the |
Hi @snaselj , We can see here, in the following lines, what role is being referred: I do not have this ipam.role in the exported Netbox json dump, I just see references to it. And it is not the only one, there are other ipam.roles being referred that are not in the dump. Should it be in the dump? Is there anything we can change in the export command to have it? I'm using the following command (similar to yours, just added exclude image attachment): |
This issue is fixed now, thanks for your help @helderbetiol . Let me know pls. whether it works for you or is there any other blocker. |
|
Hi @snaselj , Thanks for the fix, it worked for that issue, now I'm facing an issue with extras.customfields with empty labels ("label": ""): This is the data from the netbox dump: |
Thanks for the detailed report @helderbetiol , fixed. |
|
Hi @snaselj , I'm having a new issue: In our netbox dump, I see that this is what we are trying to create here: We also have a dcim.devicerole with the same name in the dump, I suppose this is why we have this issue: I can see that last one being successfully created in the logs prior to the error: Is this something that could be fixed? Or maybe I can try to change the name of one of them. |
Will fix it and let you know @helderbetiol . |
Hi @helderbetiol , The issue with duplicit role names is fixed now, thanks for your reports. |
snaselj
force-pushed
the
u/snaselj-napps-204-netbox-v3.7
branch
from
80d9d12 to
220646e
Compare
|
Cleaned up this pull request to contain only changes related to NetBox 3.7 @glennmatthews @chadell |
Hi @snaselj , I got a new issue, could you please check if it can be fixed? |
glennmatthews
approved these changes
Feb 28, 2024
docs/user/app_use_cases.md
Outdated
Comment on lines
65
to
72
| --indent=2 \ | ||
| --natural-primary \ | ||
| --natural-foreign \ | ||
| --exclude extras.ObjectChange \ | ||
| --exclude=extras.ObjectChange \ | ||
| --exclude=extras.Report \ | ||
| --exclude=extras.Script \ | ||
| --exclude=django_rq.Queue \ | ||
| --output=/tmp/netbox_data.json |
There was a problem hiding this comment.
Does the list of --exclude options need to differ by NetBox version? Or do the same options apply to all supported versions at this time?
There was a problem hiding this comment.
Yes it differ. Prior to NetBox 3.7, it was possible to export everything. With NetBox 3.7 I was facing issues dumping mentioned models, so that's why I added these excludes.
There was a problem hiding this comment.
Ok, so do we need to document different instructions for different NetBox versions?
There was a problem hiding this comment.
There is the following sentence already:
It's possible to exclude other models from the export, but the command provided above is a good starting point.
Not sure whether adding multiple examples for multiple NetBox versions wouldn't make it less readable. I would rather keep it as is and let the user figure it out based on their version and preference. WDYT?
There was a problem hiding this comment.
How should the user figure it out? Do we get clear messages if they exclude the wrong models or fail to exclude the correct models?
There was a problem hiding this comment.
Reverted --excludes. Added an error message example and how to handle that.
Hi @helderbetiol , Not sure, how to fix this. I just added a workaround, to collect and skip Nautobot instances that fails |
The order of fields in identifiers matters, as those are used to lookups.
glennmatthews
approved these changes
Feb 29, 2024
Hi @helderbetiol, @hchybz I've merged this pull request. If you have any issues, don't hesitate to open an issue in this repository and mark me there, thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes: NaN
What's Changed
v3.7tests.To Do
To see changes between 3.6 and 3.7 NetBox versions it's possible to compare the following files, e.g. with git:
https://github.com/nautobot/nautobot-app-netbox-importer/blob/u/snaselj-napps-204-netbox-v3.7/nautobot_netbox_importer/tests/fixtures/3.6/summary.txt
https://github.com/nautobot/nautobot-app-netbox-importer/blob/u/snaselj-napps-204-netbox-v3.7/nautobot_netbox_importer/tests/fixtures/3.7/summary.txt
It's also possible to compare structutred JSON summaries as well.