Duplicate Webhooks Being Generated

[adparis99]

Hello,

I have a general workflow I've been testing that goes something like this:

• Interface is updated in NetBox
• Webhook is fired off to a job in Jenkins
• Jenkins runs an Ansible script to deploy the configuration to the appropriate device/interface

I've been playing with it so I can work out potential bugs, and one thing I've found is NetBox seems to send an inconsistent number of webhooks. For example, if I update 4 interfaces on a device at once, Jenkins will end up creating 8 jobs...or 5 jobs, or 6. If I update just one interface, one job is created as expected. I'm curious if there's a way to verify that this is a problem with NetBox? Is there a log of past webhooks somewhere?

[kkthxbye-code]

In the admin panel under background tasks. If you are expecting a bulk edit of multiple interfaces to only generate one we hook request, that is indeed a wrong assumption. One webhook request will be generated for each individual object that is edited.

[adparis99]

Hey there, thanks for the reply. Like you mention, my expectation is indeed that one webhook gets generated per edited object. The weird thing is that multiple webhooks seem to be being generated on some objects, at least based on the number of job builds that Jenkins is creating. There was an edit I made earlier to 4 interfaces that generated 8 builds in Jenkins, 2 for each object. But it seems to change; I've seen edits of those same 4 interfaces generate 5, 6, and even 10 builds.

Strangely, I navigated to Background Tasks and I see the following:

I usually login with an LDAP account, so I tried the local administrator in case it was a permissions issue, but I saw the same. Any thoughts on why I'm not seeing anything there? Is there somewhere where I need to manually enable these logs?

[candlerb]

There is a log of past webhooks visible under (User) > Admin > System:Background Tasks > Finished Jobs. Then at least you can see how many webhook requests were created and delivered - click on the count to see the individual jobs. Or maybe some were rejected by Jenkins; there's a separate count for Failed Jobs.

What do you mean "at once"? Do you mean that you are hitting the REST API with multiple requests for the same device concurrently? This isn't a great idea in general, because Netbox's Django model validation cannot enforce consistency between two overlapping transactions (except in cases where the validation is done by the database's own referential integrity). For example, you can get Netbox to allocate duplicate IP addresses this way.

EDIT: having said this: if you are making separate updates to distinct interfaces then there probably shouldn't be a problem, since these are separate objects.

If you perform a single bulk update operation (where you supply a single PATCH payload with a JSON list of interfaces to update) then I think it should be safe. Writing a Custom Script which does all its work within one transaction should also be safe - you must remember to take a snapshot of each object for change logging, but I don't know if that's required for webhooks too.

[adparis99]

Hey there, thanks for the reply. As you suggested in your edit, by at once I mean I'm going into the NetBox GUI, selecting a handful of interfaces, and then editing them all at once. But Jenkins seems to be getting multiples/duplicates of some of them, and it seems to be random. Strangely, I navigated to Background Tasks and I see the following:

I usually login with an LDAP account, so I tried the local administrator in case it was a permissions issue, but I saw the same. Any thoughts on why I'm not seeing anything there? Is there somewhere where I need to manually enable these logs?

[kkthxbye-code]

Succesful webhooks are cleared after a while. Just redo the replication steps and check again.

[adparis99]

Thanks for the tip. I updated the tagged and untagged VLAN for 4 interfaces and it did indeed create 8 webhooks. I think I'm getting the pattern; it seems like a webhook is created for each change that is made to the interface, as opposed to each object. I tested again changing only the untagged VLAN for the group, and it only generated 4 webhooks. Could you confirm whether that is expected behavior or not?

EDIT: Just as an update, I've found that this isn't always the case. It's pretty strange.
When I attempt to update both the untagged and tagged VLAN of a group of interfaces, only the tagged VLAN gets saved; in that situation, two webhooks are created, but they are both incorrect as they don't contain the newly set untagged VLAN. When I attempt to update a group of interfaces and change only the tagged or the untagged VLAN, the update is made properly and an appropriate number of webhooks are generated. I'm on v3.4.2 for what it's worth. I'm going to continue testing, look at known bugs, and see if I can find a solution or perhaps need to create a bug report.