Skip to content

Commit

Permalink
Fixes behaviour of example dag tests for main/other branches (#43273) (
Browse files Browse the repository at this point in the history
…#43307)

While the #43260 attempted to address the problem where example
dag importability tests should skip provider tests on non-main,
it did not actually solve the problem.

While debugging it, it turned out that since #42505, the provider
tests were not executed in main "at all" - the "providers"
directory was not included in the list of places to check for
the example dags (they were in "airflow" in v2-10-test") this is
why it "looked like" the solution worked in "main".

This PR fixes both problems:

* brings back importability of provider's example_dags in main branch
* properly excludes the providers examples in non-main branch
  • Loading branch information
potiuk authored and utkarsharma2 committed Oct 24, 2024
1 parent 7b37541 commit 5b13431
Showing 1 changed file with 13 additions and 8 deletions.
21 changes: 13 additions & 8 deletions tests/always/test_example_dags.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,14 @@ def get_python_excluded_providers_folders() -> list[str]:


def example_not_excluded_dags(xfail_db_exception: bool = False):
example_dirs = ["airflow/**/example_dags/example_*.py", "tests/system/**/example_*.py"]
example_dirs = [
"airflow/**/example_dags/example_*.py",
"tests/system/**/example_*.py",
"providers/**/example_*.py",
]

default_branch = os.environ.get("DEFAULT_BRANCH", "main")
include_providers = default_branch == "main"

suspended_providers_folders = get_suspended_providers_folders()
current_python_excluded_providers_folders = get_python_excluded_providers_folders()
Expand All @@ -135,14 +142,7 @@ def example_not_excluded_dags(xfail_db_exception: bool = False):
for provider in current_python_excluded_providers_folders
]
providers_folders = tuple([AIRFLOW_SOURCES_ROOT.joinpath(pp).as_posix() for pp in PROVIDERS_PREFIXES])

default_branch = os.environ.get("DEFAULT_BRANCH", "main")
include_providers = default_branch == "main"

for example_dir in example_dirs:
if not include_providers and "providers/" in example_dir:
print(f"Skipping {example_dir} because providers are not included for {default_branch} branch.")
continue
candidates = glob(f"{AIRFLOW_SOURCES_ROOT.as_posix()}/{example_dir}", recursive=True)
for candidate in sorted(candidates):
param_marks = []
Expand All @@ -167,6 +167,11 @@ def example_not_excluded_dags(xfail_db_exception: bool = False):
param_marks.append(pytest.mark.xfail(reason="Expected DB call", strict=True))

if candidate.startswith(providers_folders):
if not include_providers:
print(
f"Skipping {candidate} because providers are not included for {default_branch} branch."
)
continue
# Do not raise an error for airflow.exceptions.RemovedInAirflow3Warning.
# We should not rush to enforce new syntax updates in providers
# because a version of Airflow that deprecates certain features may not yet be released.
Expand Down

0 comments on commit 5b13431

Please sign in to comment.