Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve kaggle system call #2892

Merged
merged 2 commits into from
Jul 3, 2024
Merged

Improve kaggle system call #2892

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4c8c8da
Use subprocess rather than os.system
nvoxland Jul 1, 2024
6f7a461
Use subprocess rather than os.system
nvoxland Jul 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions deeplake/auto/unstructured/kaggle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import deeplake
import glob
import os
import subprocess
from deeplake.util.exceptions import (
ExternalCommandError,
KaggleMissingCredentialsError,
Expand All @@ -14,10 +15,10 @@
from typing import Optional


def _exec_command(command):
out = os.system(command)
if out != 0:
raise ExternalCommandError(command, out)
def _exec_command(command: list[str]):
out = subprocess.run(command)
if out.returncode != 0:
raise ExternalCommandError(" ".join(command), out.returncode)

Check warning on line 21 in deeplake/auto/unstructured/kaggle.py

View check run for this annotation

Codecov / codecov/patch

deeplake/auto/unstructured/kaggle.py#L19-L21 

Added lines #L19 - L21 were not covered by tests


def _set_environment_credentials_if_none(kaggle_credentials: Optional[dict] = None):
Expand Down Expand Up @@ -90,7 +91,7 @@
raise ValueError(
"Invalid Kaggle dataset tag. Example: 'coloradokb/dandelionimages'"
)
_exec_command("kaggle datasets download -d %s" % (tag))
_exec_command(["kaggle", "datasets", "download", "-d", tag])

Check warning on line 94 in deeplake/auto/unstructured/kaggle.py

View check run for this annotation

Codecov / codecov/patch

deeplake/auto/unstructured/kaggle.py#L94 

Added line #L94 was not covered by tests

for item in os.listdir():
if item.endswith(".zip"):
Expand Down
Loading