Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Expand on Multiple Parameters Not Supported in Dynamic Task Mapping #301

Closed
su-four opened this issue Dec 1, 2024 · 0 comments · Fixed by #314
Closed

[Bug] Expand on Multiple Parameters Not Supported in Dynamic Task Mapping #301

su-four opened this issue Dec 1, 2024 · 0 comments · Fixed by #314
Assignees
Labels
bug Something isn't working triage-needed

Comments

@su-four
Copy link

su-four commented Dec 1, 2024

DAG Factory version

0.20.0

airflow version

2.10

Python version

3.12

Deployment

Docker-Compose

What happened?

The dag-factory library currently does not support the expand feature for multiple parameters in Airflow's dynamic task mapping. When specifying multiple parameters in the expand field, such as op_args using multiple outputs from upstream tasks, the second parameter is not parsed or replaced with the actual task output.

Expected Behavior:
The dag-factory library should replace both third.output and forth.output with the actual outputs from the respective tasks (third and forth) and pass them correctly to the task five. how airflow do this

Actual Behavior:
Only the first parameter (third.output) is parsed and replaced. The second parameter (forth.output) remains unchanged and is not recognized as a valid dynamic mapping.

How to reproduce

For example, when defining an operator in the YAML configuration like this:

five:  
  operator: "airflow.operators.python.PythonOperator"  
  python_callable_name: "five"  
  python_callable_file: "/funcs/test.py"  
  expand:  
    op_args: [third.output, forth.output]  
  dependencies: [third, forth]  

You can observe that the second parameter (forth.output) is not replaced or parsed correctly. The expected behavior is for both third.output and forth.output to resolve to the actual outputs of the third and forth tasks, respectively. However, only third.output is parsed and replaced, while forth.output remains unchanged, causing the operator to fail or behave unexpectedly.

Anything else :)?

This issue impacts workflows that rely on passing multiple outputs to a single downstream task, a key feature of Airflow's dynamic task mapping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage-needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants