Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New import_netbox_objectchange_json command #40

Merged
merged 30 commits into from
Apr 28, 2021

Conversation

chadell
Copy link
Contributor

@chadell chadell commented Apr 26, 2021

Addresses issue #39

In this PR:

  • Moving some shared helper methods to nautobot_netbox_importer/command_utils.py to be reused by both commands (it was not possible to move to utils.py due circular dependency from adapters
  • Adding logic to import ObjectChanges from Netbox to Nautobot, using the proper ContentType references, the user and the creation date.
  • There is a dryrun mode to run the command without changing the database

TODO

  • Add testing
  • Add documentation

EXAMPLE

nautobot-server import_netbox_objectchange_json imp/script/netbox_no_changelog.json imp/script/netbox_only_changelog.json 2.10.3 --dry-run
12:54:05 info     Loading NetBox JSON data into memory...
  filename: {'imp/script/netbox_no_changelog.json'}
12:54:07 info     JSON data loaded into memory successfully.
12:54:07 info     Creating NetBox ContentType mapping...
12:54:07 info     Creating Nautobot ContentType mapping...
12:54:07 info     Loading ObjectChange NetBox JSON data into memory...
  filename: {'imp/script/netbox_only_changelog.json'}
  0%|                                                                                                   |      0/726784 [00:00]/opt/nautobot/lib/python3.8/site-packages/cacheops/redis.py:19: RuntimeWarning: The cacheops cache is unreachable! Error: Client sent AUTH, but no password is set
  warnings.warn("The cacheops cache is unreachable! Error: %s" % e, RuntimeWarning)
 50%|█████████████████████████████████████████████████▎                                                 | 361843/726784 [16:41]13:11:00 warning  ('ip_health_check', 'healthcheckrecord') key is not present in Nautobot
100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 726784/726784 [17:16]
13:11:35 info     Processed 726784 in this run.

Copy link
Contributor

@glennmatthews glennmatthews left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the pull request! I like the approach here a lot.

self.logger, _ = initialize_logger(options)

self.logger.info("Loading NetBox JSON data into memory...", filename={options["json_file"].name})
no_objectchange_data = json.load(options["json_file"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

netbox-docker, by default, prepends a few lines of text log output to any manage.py command, including manage.py dumpdata, which causes the resulting file to not be valid JSON until/unless you manually delete those lines with a text editor. I mention this just to suggest that you add a try...except ValueError around each of the json.load calls so that if the file isn't valid JSON, you can handle it gracefully with an appropriate error message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@glennmatthews
Copy link
Contributor

What's the story behind the changes to poetry.lock? I don't see any corresponding changes to pyproject.toml.

@chadell
Copy link
Contributor Author

chadell commented Apr 27, 2021

What's the story behind the changes to poetry.lock? I don't see any corresponding changes to pyproject.toml.

terror story... I will rollback it

@chadell chadell changed the title [WIP] New import_netbox_objectchange_json command New import_netbox_objectchange_json command Apr 28, 2021
@chadell chadell marked this pull request as ready for review April 28, 2021 07:58
@chadell chadell requested a review from glennmatthews April 28, 2021 11:20
Copy link
Contributor

@glennmatthews glennmatthews left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates!

@chadell chadell requested a review from glennmatthews April 28, 2021 13:54
@chadell chadell requested a review from glennmatthews April 28, 2021 18:02
@glennmatthews glennmatthews merged commit 777d2db into nautobot:develop Apr 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants