feat(platform, blocks): Webhook-triggered blocks

[Pwuts]

• Resolves Implement GitHub "on pull request" block #8352

Changes 🏗️

• feat(blocks): Add GitHub Pull Request Trigger block

feat(platform): Add support for Webhook-triggered blocks

• ⚠️ Add PLATFORM_BASE_URL setting

• Add webhook config option and BlockType.WEBHOOK to Block

  • Add check to Block.__init__ to enforce type and shape of webhook event filter
  • Add check to Block.__init__ to enforce payload input on webhook blocks
  • Add check to Block.__init__ to disable webhook blocks if PLATFORM_BASE_URL is not set

• Add Webhook model + CRUD functions in backend.data.integrations to represent webhooks created by our system

  • Add IntegrationWebhook to DB schema + reference AgentGraphNode.webhook_id
    • Add set_node_webhook(..) in backend.data.graph

• Add webhook-related endpoints:

  • POST /integrations/{provider}/webhooks/{webhook_id}/ingress endpoint, to receive webhook payloads, and for all associated nodes create graph executions
    • Add Node.is_triggered_by_event_type(..) helper method
  • POST /integrations/{provider}/webhooks/{webhook_id}/ping endpoint, to allow testing a webhook
  • Add WebhookEvent + pub/sub functions in backend.data.integrations

• Add backend.integrations.webhooks module, including:

  • graph_lifecycle_hooks, e.g. on_graph_activate(..), to handle corresponding webhook creation etc.
    • Add calls to these hooks in the graph create/update endpoints
  • BaseWebhooksManager + GithubWebhooksManager to handle creating + registering, removing + deregistering, and retrieving existing webhooks, and validating incoming payloads

Other improvements

• fix(blocks): Allow having an input and output pin with the same name
• fix(blocks): Add tooltip with description in places where block inputs are rendered without NodeHandle
• feat(blocks): Allow hiding inputs (e.g. payload) with SchemaField(hidden=True)
• fix(frontend): Fix MultiSelector component styling
• feat(frontend): Add AlertDialog UI component
• feat(frontend): Add NodeMultiSelectInput component
• feat(backend/data): Add NodeModel with graph_id, graph_version; GraphModel with user_id
  • Add make_graph_model(..) helper function in backend.data.graph
• refactor(backend/data): Make RedisEventQueue generic and move to backend.data.execution
• refactor(frontend): Deduplicate & clean up code for different block types in generateInputHandles(..) in CustomNode
• dx(backend): Add MissingConfigError, NeedConfirmation exception

How it works

• When a graph is created, the on_graph_activate and on_node_activate hooks are called on the graph and its nodes
• If a webhook-triggered node has presets for all the relevant inputs, on_node_activate will get/create a suitable webhook and link it by setting AgentGraphNode.webhook_id
  • on_node_activate uses webhook_manager.get_suitable_webhook(..), which tries to find a suitable webhook (with matching requirements) or creates it if none exists yet
• When a graph is deactivated (in favor of a newer/other version) or deleted, on_graph_deactivate and on_node_deactivate are called on the graph and its nodes to clean up webhooks that are no longer in use
• When a valid webhook payload is received, two things happen:
  1. It is broadcast on the Redis channel webhooks/{webhook_id}/{event_type}
  2. Graph executions are initiated for all nodes triggered by this webhook

TODO

• Write developer documentation for webhook-triggered blocks #8537
• Render dict[str, bool] block input as collapsible multi-select #8538
• On deletion of integration credentials -> delete all associated webhooks first #8357
• UX improvements on GitHubPullRequestTriggerBlock #8737
• Prevent use of webhook-triggered blocks if PLATFORM_BASE_URL is not set #8738
• Test test test!
• Add note on repo input of webhook blocks that the credentials used must have the right permissions for the given organization/repo
• Implement proper detection and graceful handling of webhook creation failing due to insufficient permissions. This should give a clear message to the user to e.g. "give the app access to this organization in your settings".

Follow-up

• Implement status indicator for webhook-triggered nodes #8743
• Implement mechanism to enable/disable webhook-triggered agents/blocks #8671
• Improve traceability of webhook-triggered graph executions #8554
• Store incoming webhook payloads and make them viewable #8742
• Implement test mechanism for webhook-triggered blocks #8744
• dx: Reduce number of files needed for OAuth and/or Webhook-triggered blocks #8735

[ntindle]

What is an event filter input? What does it do? Do I need one? what if I can't filter my webhooks

[Pwuts]

It currently is required, as it fits all the existing implementations. Making it optional wouldn't be hard, but also currently isn't necessary.

The angle to this PR is: it's the first version of the webhook system and will be iterated on further as we add support for more providers.

[Pwuts]

I've amended the description. Making this input optional can be done when it becomes necessary in a follow-up PR.

[ntindle]

Can this be a pydantic model?

[Pwuts]

In theory, sure. I'm not sure the existing implementation of webhook ingress endpoint + executor supports it though.

[ntindle]

How do we handle when there is no provider api?

[ntindle]

(use this to make follow up ticket)

[Pwuts]

That's a question I left to answer when we actually have to deal with that

[ntindle]

Yeah this is kinda a weirdly restricting thing

[Pwuts]

The restriction makes the rest of the implementation easier. Also, this structure allows adding individual descriptions to all the options, which I can definitely see being necessary for good UX if event names are not self-explanatory.

[ntindle]

This makes a heavily flawed assumption of all webhooks must be registered via an API. Provide a simple webhook manager that does not require this.

[ntindle]

#8750 is my attempt at this for context

[Pwuts]

Let's fix that in your PR as a follow-up.