Distributed executor?

[aadrian]

Hi,

Would it be possible to have a distributed executor (or independent of the GUI) so that one workflow can span more than one server?

Thank you.

[yohamta]

Hi 👋

Unfortunately it's currently designed for only one GUI & one scheduler (works as a worker as well).
We'd like to add the multiple worker functionality after some data schema refactoring.

Thanks!

[aadrian]

...after some data schema refactoring

Any details what this might involve?

I could not find any current "schema file" or DB Model - the "data model" seems distributed across packages (I'm not sure what really belongs to it, in order to see what might need refactoring)

[yohamta]

You're right! Dagu doesn't have database schema and writing everything to JSON files for now.

I was lightly thinking that we need to have database to have multiple workers, but yeah it's totally possible to have mulitple workers without having any database.

Dagu server has REST(ish) API endpoint so adding the worker addresses to the scheduler configuration file will make it possible to implement the functionality fairly easily I think.

[yohamta]

I think in the future having database is great but for now it's not necessary IMO.

I am totally fine with adding the multiple worker functionality while keeping the current JSON file database mechanism as is.

[yohamta]

I think this document would be interesting.
Airflow's Celery Executor

[yohamta]

Here's my initial idea about the worker distribution functionality.

[aadrian]

This looks great !

Would it be a little simpler if the GUI server would also dispatch the execution? Since the GUI still needs to interact with the workers anyway?
In the current implementation diagram, the scheduler also does not seem interact directly with the worker.

[yohamta]

Thanks! Yeah, you're right. GUI server and scheduler can be a single process.
I was thinking to add the functionality to the server command by adding --scheduler flag.

[yohamta]

In the current implementation diagram, the scheduler also does not seem interact directly with the worker.

I forgot to add the arrow and fixed it, thanks! =D

[yohamta]

Implementing the functionality would involve these tasks.

• Add Workers option to the server configuration
• Update Scheduler process to interact with Worker servers
• Implement functionality for collecting workflow history data from workers

[aadrian]

Would the GUI/API call the Worker Servers (A), or the Worker Servers call the central GUI/API only (B) , or Bidirectional (C) ?

Implementing the functionality would involve these tasks.
Add Workers option to the server configuration

In this mode, the GUI would not need to work or be rendered, right ?
(To have a "local sub-GUI" for each Worker Server might be an overkill.)

Implement functionality for collecting workflow history data from workers

This would depend on the direction of communication I suppose.
In case of A the Worker Servers would not even need an exposed port, since they would act as clients for the GUI/API ?

[yohamta]

Would the GUI/API call the Worker Servers (A), or the Worker Servers call the central GUI/API only (B) , or Bidirectional (C) ?

I think the GUI server, or maybe even a separate data collection process, will have to call the Worker Servers' API to gather log data.

In this mode, the GUI would not need to work or be rendered, right ? (To have a "local sub-GUI" for each Worker Server might be an overkill.)

You're right! The Worker servers don't need a GUI themselves, even though one's included by default right now. Not sure if we should take it out just yet. 🤔

This would depend on the direction of communication I suppose. In case of A the Worker Servers would not even need an exposed port, since they would act as clients for the GUI/API ?

I might be getting confused here. I was under the impression that in scenario A, the Worker Servers would still need to expose a port to the GUI server for API calls. Can you help me understand what you have in mind?

[aadrian]

I might be getting confused here. I was under the impression that in scenario A, the Worker Servers would still need to expose a port to the GUI server for API calls. Can you help me understand what you have in mind?

Sorry for the confusion - you are right: I was trying to point out that scenario "B" (not "A" - as I mistakenly wrote) seems like the simplest one to implement and also to secure, since it would be a simple "client/server" architecture.

[yohamta]

No worries! You're right, I see what you mean now. Scenario "B" - with the workers pushing the log to the server - does seem like the simplest and most secure way to go. Great catch! That actually sounds like a much better approach 😃

[yohamta]

Here's the revised version of the architecture.

[aadrian]

the revised version of the architecture.

This looks great !

[yohamta]

Thanks!!

[gedw99]

Hey all !!

This is a great system.

You can do most of whats in that Architecture diagram and have a global super cluster using just NATS Jetstream with almost no changes to DAGU.

The NATS Object store is an easy way to take a system that uses a File system for its state and make it globally distributed. When a New Dag is put in, all other servers will instantly know, and can react to that. This is because the Object store is globally reactive.

The NATS KV store allows DAGU to be parametric also. so that when a value input or output in Dagu changes then other dags on other servers will be told and can rerun. You your dags are distributed and parametric and reactive.

Linking the existing DAGS to use the NATS KV woudl just be a matter of adding it as a type into the existing Dagu system.

The DAFGI REST API and all the existing work can be unchanged. The CONFIG and BASE configs etc will just work. It's just a File System after all. The Object Store will ensure this is kept in sync globally.

You can easily add tasks like Dagu into the system and NATS can ensure there is only one running at a time of all the global servers.

https://github.com/choria-io/asyncjobs does this using NATS can I think can easily work with Dagu or be integrated.

https://github.com/choria-io/appbuilder/blob/main/example/sample-app.yaml is very similar to Dagu, and they use asyncjobs with NATS to runs jobs described by appbuilder.

Doing Distributed Jobs is inherently difficult to get right and can take years to get stable.

NATS and these high level NATS systems can really work well with DAGU, and gives you a global control plane on the existing Dagu data plane. ...