Reduces flushes during commit/checkout #1381
Conversation
| @@ -403,7 +403,8 @@ def commit(self, message: Optional[str] = None) -> None: | |||
| str: the commit id of the stored commit that can be used to access the snapshot. | |||
| """ | |||
| commit_id = self.version_state["commit_id"] | |||
| try_flushing(self) | |||
There was a problem hiding this comment.
we already flush after copying metas inside commit function in version_control.py so this wasn't needed
| @@ -403,7 +403,8 @@ def commit(self, message: Optional[str] = None) -> None: | |||
| str: the commit id of the stored commit that can be used to access the snapshot. | |||
| """ | |||
| commit_id = self.version_state["commit_id"] | |||
| try_flushing(self) | |||
| initial_autoflush = self.storage.autoflush | |||
There was a problem hiding this comment.
always disable autoflush during commit and enable it back at the end if required
| @@ -426,7 +428,8 @@ def checkout(self, address: str, create: bool = False) -> str: | |||
| Returns: | |||
| str: The commit_id of the dataset after checkout. | |||
| """ | |||
| try_flushing(self) | |||
There was a problem hiding this comment.
we can remove this call as we flush during copy_metas during checkout in version_control.py if a new branch is being created, otherwise if we're checking out to an existing branch/commit, explicit flushes have been added now.
Codecov Report
@@ Coverage Diff @@
## main #1381 +/- ##
==========================================
- Coverage 92.83% 92.79% -0.05%
==========================================
Files 174 174
Lines 12749 12757 +8
==========================================
+ Hits 11836 11838 +2
- Misses 913 919 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
🚀 🚀 Pull Request
Checklist:
coverage-rateupChanges
Makes commits and checkout to new branches faster