Run DAG on schedule unless it has been run since last scheduled run

[chrishoage]

One thing I have not been able to figure out is how to prevent a DAG from running if it has been run before the scheduler would typically run the job.

I'm wondering if there is a cleaner way than to use the HTTP executor to get the finishedAt attribute, get the current time from shell, parse the cron string, and do the math to see if it's run since the last run?

Some questions:

• are any of the DAG details exposed via the environment available to the DAG?
• is there a DAG setting available to achieve this behavior that I missed in the documentation?
• aside from creating a collection of scripts and exposing them on the PATH is there a way to share "DAG functions" between DAGs?
• is there any environment variables set for scripts or commands that dagu invokes that would let the script know which DAG is invoking it?

I would like to share this "delay run if run before scheduled run" between DAGs, and it seems like it's going to be a little cumbersome.

I believe the script will need to:

• be told what DAG is executing it directly in a step
• fetch the detail status of the DAG
• read the DAG file to determine the schedule
• parse the cron line into a schedule
• compare the last finish with the next scheduled run if the DAG is not in a running state
• use the output of this command as a precondition for continuing the DAG

Is there a simpler way?

[yohamta]

Hey Chris,

Thanks for the detailed questions! You've brought up some really good points about how we can make DAG management in Dagu even better.

Preventing a DAG from rerunning if it has already completed is a smart idea. I'm thinking of adding a runOnlyOnSchedule flag at the DAG level. If you set that to true, Dagu will automatically check the last successful run time against the schedule and skip the current run if it's not needed. This should make your current workaround unnecessary.

Regarding your other questions:

• DAG Details: We actually already have some special environment variables documented here. We could definitely add more, like DAG_SCHEDULE and DAG_LAST_SUCCESSFUL_RUN, to give your tasks more info. What do you think?

• Sharing DAG Functions: Right now, we support sub-DAGs (check out the docs here). We know it's not quite the same as reusable functions, but it might help in some cases. We're definitely thinking about better ways to modularize DAGs, so your input is super helpful. One limitation right now is that a single DAG can't run more than once concurrently, but I'm planning to change that.

• Knowing Which DAG is Running: The DAG_REQUEST_ID environment variable is available during execution, which can be useful for tracking.

That runOnlyOnSchedule flag seems particularly helpful, so I'm definitely going to look into implementing that. Thanks again for the great feedback! Let us know what you think.

[chrishoage]

We actually already have some special environment variables documented here. We could definitely add more, like DAG_SCHEDULE and DAG_LAST_SUCCESSFUL_RUN, to give your tasks more info. What do you think?

I've found these which lead me to wonder if there may be more undocumented ones.

I do wonder however since jq support is a first class executor if it would be simpler to simply expose the details as DAG_DETAILS_JSON

Then steps could be written like this

steps:
  - name: get value
    executor: jq
    command: '{( .schedule }}'
    script: "echo $DAG_DETAILS_JSON
   output: DAG_SCHEDULE

That isn't to say having bespoke environment variables wouldn't be handy for more frequently accessed values. It would save needing a step simply to fetch the details.

we support sub-DAGs (check out the docs here). We know it's not quite the same as reusable functions, but it might help in some cases.

sub-DAGs are one of the primary ways I am planning on using this tool.

I have a media ingestion script that is currently a monolith, but there's no reason why it could not be split up so parts of this script could be run on demand. I plan to break the major parts of this script into DAGs and have a meta-DAG that runs them together on a schedule.

Knowing Which DAG is Running: The DAG_REQUEST_ID environment variable is available during execution, which can be useful for tracking.

There doesn't appear to be a straight forward way from the API to know what DAG is running from the request id alone. It seems it would require hitting GET /api/v1/dags for the list of DAGs the searching this list for the request ID.

We would need DAG_NAME in order to easily look up the detail of a DAG from a script via the API

That runOnlyOnSchedule flag seems particularly helpful, so I'm definitely going to look into implementing that.

Thank you! To add more context as to why runOnlyOnSchedule would be useful:

With my media ingestion script mentioned above at the end I run a backup command. This backup command is expensive computationally, and also wakes up spinning hard drives that I would prefer to keep spun down if possible.

My media ingestion script doesn't run every day however, and if there is nothing to ingest I don't wish to run the backup command.

However I would still like to run the backup command periodically.

My goal is to have the backup command in a DAG on a schedule, and have my meta-DAG run this backup command DAG. However the next time the schedule triggers I don't want the backup DAG to run if it ran since the last time it was supposed to run on schedule.

For example:

• backup DAG is scheduled to run every 3 days.
• backup DAG is triggered manually 1.5 days after its normally scheduled run
• backup DAG schedule timer fires, but it has already run recently, so it doesn't run.
• 3 days pass
• backup DAG schedule timer fires, it has not run in the last 3 days, so it runs.

Thanks again for the great feedback! Let us know what you think.

Thank you for this software! I'm still evaluating it but looks like it is going to fit my needs. I have further ideas, but I'll wait to supply these until I have done more than read the docs and the code 😅

[yohamta]

Thanks for the detailed response and insightful suggestions! This is super helpful.

Just to clarify, DAG_SCHEDULE and DAG_LAST_SUCCESSFUL_RUN were just ideas. Your DAG_DETAILS_JSON suggestion is really clever! It would definitely simplify things and provide a lot of flexibility. I'm going to look into that more closely.

I'm also glad to know your plan on using sub-DAGs! That's great feedback.

You're spot on about DAG_REQUEST_ID. Adding DAG_NAME as an environment variable makes perfect sense. I'll add that to the list.

The extra context on runOnlyOnSchedule is very helpful. The backup DAG example perfectly illustrates the need. I completely understand wanting to avoid unnecessary runs, especially for resource-intensive tasks. This is definitely something I want to looking into.

I really appreciate your enthusiasm and detailed feedback! It's incredibly helpful as we develop Dagu. I'd love to hear any other ideas you have as you keep exploring. Every bit of feedback helps!

[chrishoage]

Unfortunately it turns out I won't be able to use dagu after all.

My use case for Dagu is running in a container and orchestrating a collection of tasks, some of which are related.

However all my tasks are broken into their own docker containers for ease of maintenance.

I discovered as I was trying to actually deploy my tasks that Dagu does not support environment variables / interpolation in anything other than a command step. None of the executor steps appear to be able to use environment variables.

I am off to look for other solutions, but some suggestions for the future which would make this tool amazing.

• Use something like go templates for dags which can be processed when loaded. This would allow for the use of env vars (or even more powerful concepts) in the all dags including executors. (E.g. something like {{ .MY_ENV }} in any part of the yaml)
• Support exec in the docker executor. When running in a container it would be useful to be able to attach to an existing container and run a command (like ofelia or compose-scheduler)
• Handling of params is currently rough at best. I'd like to see something like go-task
• Have the ability to not have all of the "job" output from running a sub dag appear in the output of the parent dag

I think ultimately what I wish existed is a task runner that is as powerful as go-task, with the ability to schedule tasks, and have a web ui to monitor the task runs as well as manually invoke a task out of schedule.

Dagu is very close but the limitations I ran up against for docker ended up being show stoppers.

Thank you for this wonderful project! I'll keep an eye on it to see if it one day will suit my needs.

[yohamta]

Thank you for your detailed feedback and for trying out Dagu! I really appreciate you taking the time to outline the limitations you encountered and suggest improvements. It's incredibly valuable.

The suggestion of using Go templates for DAG definitions is excellent and would address this elegantly. It's definitely something I'll consider for the next version.

Supporting exec in the Docker executor is another crucial feature for containerized workflows, enabling interaction with existing containers as you described. I agree that this would significantly enhance Dagu's usability in containerized environments.

The feedback on parameter handling and "job" output from sub-DAGs is also well-taken. Improving the parameter handling experience and providing more control over output verbosity are important for usability. go-task's parameter handling is indeed a good model to consider.

I understand that the current limitations are showstoppers for your use case. I appreciate your kind words and hope that future developments will make Dagu a suitable solution for you down the line. I'll keep these points in mind as I continue to work on the project. Please feel free to open issues on the project's repository (if there is one) to keep the conversation going and track progress on these features.