HDDS-9260. Optimize OmDbInsight API to avoid redundant recursive size calculations for deleted directories during data retrieval. #5265
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… calculations for deleted directories during data retrieval.
devmadhuu
reviewed
Sep 14, 2023
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/OMDBInsightEndpoint.java
Show resolved
Hide resolved
… Integration tests to verfiy the result
|
@devmadhuu @sumitagrawl @ashishkumar50 can you please look |
devmadhuu
approved these changes
Sep 19, 2023
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/types/NSSummary.java
Outdated
Show resolved
Hide resolved
sumitagrawl
requested changes
Sep 23, 2023
There was a problem hiding this comment.
@ArafatKhan2198 I have few scenario which seems not handled,
- if sub-directory is removed, then the sub-dir size is not removed from parent -- This needs re-calculation again -- This is major issue with this implementation as needs recursive update till its parent
- NSSummary when save to DB need extra field also, may not be much problem
|
Additionally,
|
This was referenced Oct 31, 2023
This was referenced Sep 3, 2024
This was referenced Sep 9, 2024
This was referenced Sep 29, 2024
This was referenced Nov 18, 2024
Merged
This was referenced Dec 8, 2024
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In the current implementation of
omdbInsights, the calculation of sizes fordeleted directoriesoccurs dynamically when needed. The respective NSSummary object associated with a deleted directory does not retain the directory's size information. Consequently, we must repeatedly traverse the directory structure recursively to calculate its size whenever the omdbInsights endpoint is invoked.To eliminate the redundancy, we can store the calculated size within the corresponding NSSummary objects. This means that the next time we need to determine the size of a specific deleted directory, we can retrieve the stored size value directly, avoiding the need for recomputation.
It's important to note that this approach is effective for deleted directories, as their size remains constant once they are deleted. Unlike normal directories, where size can change dynamically as files are added or removed, the size of deleted directories remains constant, making it a suitable candidate for this optimization.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9260
How was this patch tested?
Integration Tests