ValidationError crash on rack import

[jathanism]

Environment

• Python version: 3.8.8
• Nautobot version: 1.0.0b1
• nautobot-netbox-importer version: 1.0.0

Expected Behavior

Full import!

I have a rather large production NetBox 2.10.3 dump (about 600 MB, ~40k devices, ~280k interfaces, ~88k cables, ~90k its, ~14k prefixes) I'm trying to import.

Observed Behavior

Crashed about ~1% of the way through import.

Traceback (most recent call last):
  File "/usr/local/bin/nautobot-server", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/cli.py", line 55, in main
    run_app(
  File "/usr/local/lib/python3.8/site-packages/nautobot/core/runner/runner.py", line 268, in run_app
    management.execute_from_command_line([runner_name, command] + command_args)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/management/commands/import_netbox_json.py", line 91, in handle
    target.sync_from(source, flags=DiffSyncFlags.SKIP_UNMATCHED_DST)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/adapters/abstract.py", line 299, in sync_from
    return super().sync_from(source, diff_class=diff_class, flags=flags)
  File "/usr/local/lib/python3.8/site-packages/diffsync/__init__.py", line 464, in sync_from
    result = syncer.perform_sync()
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 251, in perform_sync
    changed |= self.sync_diff_element(element)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 283, in sync_diff_element
    changed, modified_model = self.sync_model(model, ids, attrs)
  File "/usr/local/lib/python3.8/site-packages/diffsync/helpers.py", line 329, in sync_model
    model = self.model_class.create(diffsync=self.dst_diffsync, ids=ids, attrs=attrs)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 203, in create
    record = cls.create_nautobot_record(cls.nautobot_model(), nautobot_ids, nautobot_attrs, multivalue_attrs)
  File "/usr/local/lib/python3.8/site-packages/nautobot_netbox_importer/diffsync/models/abstract.py", line 180, in create_nautobot_record
    record.clean()
  File "/usr/local/lib/python3.8/site-packages/nautobot/extras/plugins/validators.py", line 20, in wrapper
    model_clean_func(model_instance)
  File "/usr/local/lib/python3.8/site-packages/nautobot/dcim/models/devices.py", line 586, in clean
    raise ValidationError(
django.core.exceptions.ValidationError: {'rack': ['Rack ABC03 (CENSORED) does not belong to site iad1.']}

Looking at NetBox from which the dump was exported, that particular rack is in site "ord1", so the error is seemingly accurate but I'm not clear on the context otherwise.

Steps to Reproduce

1. Receive my massive database dump "somehow"
2. nautobot-server import_netbox_json /tmp/netbox_data.json 2.10.3 and then wait
3. Crash!

[jathanism]

BTW I did try this patch:

diff --git a/diffsync/models/abstract.py b/diffsync/models/abstract.py
index 36accb9..a44216d 100644
--- a/diffsync/models/abstract.py
+++ b/diffsync/models/abstract.py
@@ -13,9 +13,11 @@ from typing import Mapping, Optional, Tuple, Union

 from diffsync import DiffSync, DiffSyncModel
 from diffsync.exceptions import ObjectNotFound
+from django.core.exceptions import ValidationError
 from django.db import models
 from django.db.utils import IntegrityError
 from pydantic import BaseModel, validator
 import structlog

 from .references import (
@@ -182,7 +184,7 @@ class NautobotBaseModel(DiffSyncModel):
             for attr, value in multivalue_attrs.items():
                 getattr(record, attr).set(value)
             return record
-        except IntegrityError as exc:
+        except (IntegrityError, ValidationError) as exc:
             logger.error(f"Error in creating {nautobot_model}: {exc}")
             return None

And it did make it further, but not totally sure that this is the correct move here.

[glennmatthews]

That's definitely a step in the right direction! Looks like I should be catching ValidationError in both create_nautobot_record and update_nautobot_record functions.

[glennmatthews]

The specific ValidationError you reported suggests that you have a Device in your source data which is assigned directly to Site "iad1" but is also assigned to a Rack that is itself assigned to Site "ord1", which Nautobot considers to be invalid. To successfully import this device you'll have to fix your source data, but I can at least update this plugin so it will report this error and continue instead of aborting.