Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace os.system('unzip file.zip') -> ZipFile.extractall() #4919

Merged
merged 2 commits into from
Sep 25, 2021

Conversation

glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Sep 25, 2021

This PR updates all YOLOv5 unzip ops from os.system() commands, which appear to cause Windows issues in a small number of cases, with the python built-in zipfile package, which should provide greater compatibility across operating systems.

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Refactoring of zip file extraction methods across multiple utility scripts.

📊 Key Changes

  • Replaced os.system unzip calls with ZipFile.extractall method in datasets.py, downloads.py, and general.py.
  • Removed command line unzip commands in favor of direct Python functions.
  • Streamlined the error handling by eliminating return codes from unzip commands.
  • Improved clarity in autodownload messages, explicitly stating the download location.

🎯 Purpose & Impact

  • Enhanced portability and security: By using Python's ZipFile module instead of system commands, code becomes more secure and works across all operating systems without relying on external unzip utilities.
  • Easier maintenance and readability: The use of built-in Python library functions ensures code is more Pythonic and easier to maintain.
  • User-friendly feedback: Clearer messages about download and extraction processes improve user experience by providing direct information on where files are saved, especially during dataset autodownload.
  • Streamlined error handling: Removal of system-dependent exit codes leads to more consistent and reliable error checking directly within Python's execution environment.

@glenn-jocher glenn-jocher linked an issue Sep 25, 2021 that may be closed by this pull request
@glenn-jocher
Copy link
Member Author

@kalenmike this will likely conflict with your open PR #4774, but also helps remove os.system() calls as you'd recommended earlier, and fixes some immediate Windows issues in #4779.

@glenn-jocher glenn-jocher self-assigned this Sep 25, 2021
@glenn-jocher glenn-jocher merged commit a64a4c8 into master Sep 25, 2021
@glenn-jocher glenn-jocher deleted the update/unzip branch September 25, 2021 15:52
glenn-jocher added a commit that referenced this pull request Sep 25, 2021
Fix for bug introduced by #4919 discovered on VOC autodownload:
```
python train.py --data VOC.yaml
```
glenn-jocher added a commit that referenced this pull request Sep 25, 2021
* Fix `root` referenced before assignment

Fix for bug introduced by #4919 discovered on VOC autodownload:
```
python train.py --data VOC.yaml
```

* Cleanup
@glenn-jocher
Copy link
Member Author

Before and After profiling results show slightly improved speeds :)

import os
import time
from pathlib import Path
from zipfile import ZipFile

file = Path('test/coco2017labels.zip')

t = time.time()
ZipFile(file).extractall(path=file.parent)
print(f'DONE PYTHON in {time.time() - t}s')

t = time.time()
os.system(f'unzip -qo {file} -d {file.parent}')
print(f'DONE BASH in {time.time() - t}s')

Returns

DONE PYTHON in 26.95448899269104s
DONE BASH in 31.251495122909546s

CesarBazanAV pushed a commit to CesarBazanAV/yolov5 that referenced this pull request Sep 29, 2021
…alytics#4919)

* Replace `os.system('unzip file.zip')` -> `ZipFile.extractall()`

* Cleanup
CesarBazanAV pushed a commit to CesarBazanAV/yolov5 that referenced this pull request Sep 29, 2021
* Fix `root` referenced before assignment

Fix for bug introduced by ultralytics#4919 discovered on VOC autodownload:
```
python train.py --data VOC.yaml
```

* Cleanup
BjarneKuehl pushed a commit to fhkiel-mlaip/yolov5 that referenced this pull request Aug 26, 2022
…alytics#4919)

* Replace `os.system('unzip file.zip')` -> `ZipFile.extractall()`

* Cleanup
BjarneKuehl pushed a commit to fhkiel-mlaip/yolov5 that referenced this pull request Aug 26, 2022
* Fix `root` referenced before assignment

Fix for bug introduced by ultralytics#4919 discovered on VOC autodownload:
```
python train.py --data VOC.yaml
```

* Cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Where I can download a small/tiny dataset to test your repo?
1 participant