HDDS-11453. OmSnapshotPurge should be in a different ozone manager double buffer batch #7188
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…uble buffer batch Change-Id: Ie70c38571370bea19941ab5bf8937311da6d67d6
hemantk-12
reviewed
Sep 11, 2024
There was a problem hiding this comment.
Thanks @swamirishi for the patch.
Left some minor comments.
...e/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
Outdated
Show resolved
Hide resolved
...e/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
Outdated
Show resolved
Hide resolved
Change-Id: I209f36c29805c8c0ceae05334967a3023a62d499
Change-Id: I849a7497783d5805609635bdcb9128909b0c13de
adoroszlai
previously requested changes
Sep 12, 2024
This reverts commit 7fb2f8e.
Change-Id: I2c4db21e763cee7a7d2603a4211d4acaf8cf77b4
Change-Id: I0eb0ad52a55bf029dc5437cecb209b7630fc709f
Change-Id: Id5ed6538c591744dc12b766c0f3ed9b8dff002bc
Change-Id: I792f0e445d237750b7f45b3142c0dec7003ce128
szetszwo
reviewed
Sep 12, 2024
szetszwo
reviewed
Sep 12, 2024
...e/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerDoubleBuffer.java
Outdated
Show resolved
Hide resolved
szetszwo
approved these changes
Sep 12, 2024
prashantpogde
approved these changes
Sep 12, 2024
aswinshakil
approved these changes
Sep 12, 2024
|
Thank you for reviewing the PR @szetszwo @prashantpogde @aswinshakil @adoroszlai @hemantk-12 |
aswinshakil
added a commit
to aswinshakil/ozone
that referenced
this pull request
Oct 1, 2024
COMMITS: 360fea5 HDDS-11494. Improve the duration option of freon ombg (apache#7246) 10d3b21 HDDS-11504. Update Ratis to 3.1.1. (apache#7257) ce46297 HDDS-11162. Improve Disk Usage page UI (apache#7214) c91f1c7 HDDS-11491. Avoid sharing clientId among deleting services (apache#7250) b0943d5 HDDS-11501. Improve logging in XceiverServerRatis (apache#7252) 55925ab HDDS-11502. Class path contains multiple SLF4J providers (apache#7255) d0ad836 HDDS-11472. Avoid recreating external access authorizer on OM state reload (apache#7238) 254db9e HDDS-11500. RootCARotationManager cancelling wrong task in notifyStatusChanged (apache#7251) 1e6e4b3 HDDS-11499. Remove redundant code from ECReconstructionCoordinator. (apache#7248) adb2821 HDDS-11490. Bump rollup to 3.29.5 (apache#7232) 189a9fe HDDS-11484. Validate javadoc in CI (apache#7245) 64a29c6 HDDS-11497. Bump commons-configuration2 to 2.11.0 (apache#7242) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snapshot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForPrefixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyInFailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snaps hot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForP refixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyIn FailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) e0060a8 HDDS-11196. Improve SCM WebUI Display (apache#6960) 22ddfb9 Revert "HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192)" 9f5bf43 HDDS-11457. Internal error on S3 CompleteMultipartUpload if parts are not specified (apache#7195) 10c47a1 HDDS-11459. Bump develocity-maven-extension to 1.22.1 (apache#7201) 50f2563 HDDS-11419. Fix waitForCheckpointDirectoryExist log message (apache#7199) a7d7e37 HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192) 5feb9ea HDDS-11453. OmSnapshotPurge should be in a different ozone manager double buffer batch (apache#7188) 703c4d5 HDDS-10984. Tool to restore SCM certificates from RocksDB. (apache#6781) d221065 HDDS-11440. Add a lastTransactionInfo field in SnapshotInfo to check for transactions in flight on the snapshot (apache#7179) e573701 HDDS-11448. Improve documentation in ContainerStateMachine (apache#7183) 0e49f7a HDDS-11449. Remove unnecessary log from client console. (apache#7184) cd251f2 HDDS-11438. Ensure DataInputBuffer is closed in OMPBHelper#convert (apache#7182) 4b47812 HDDS-11389. Incorrect number of deleted containers shown in Recon UI. (apache#7149) 0915f0b HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. (apache#7009) 0f16195 HDDS-11416. refactor ratis submit request avoid code duplicate (apache#7166) 86fe920 HDDS-11376. Improve ReplicationSupervisor to record replication metrics (apache#7140) CONFLICT: Merge conflict in hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandlerWithUnhealthyContainer.java MODIFIED: /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/DNContainerOperationClient.java /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java
aswinshakil
added a commit
to aswinshakil/ozone
that referenced
this pull request
Oct 1, 2024
COMMITS: 360fea5 HDDS-11494. Improve the duration option of freon ombg (apache#7246) 10d3b21 HDDS-11504. Update Ratis to 3.1.1. (apache#7257) ce46297 HDDS-11162. Improve Disk Usage page UI (apache#7214) c91f1c7 HDDS-11491. Avoid sharing clientId among deleting services (apache#7250) b0943d5 HDDS-11501. Improve logging in XceiverServerRatis (apache#7252) 55925ab HDDS-11502. Class path contains multiple SLF4J providers (apache#7255) d0ad836 HDDS-11472. Avoid recreating external access authorizer on OM state reload (apache#7238) 254db9e HDDS-11500. RootCARotationManager cancelling wrong task in notifyStatusChanged (apache#7251) 1e6e4b3 HDDS-11499. Remove redundant code from ECReconstructionCoordinator. (apache#7248) adb2821 HDDS-11490. Bump rollup to 3.29.5 (apache#7232) 189a9fe HDDS-11484. Validate javadoc in CI (apache#7245) 64a29c6 HDDS-11497. Bump commons-configuration2 to 2.11.0 (apache#7242) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snapshot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForPrefixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyInFailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snaps hot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForP refixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyIn FailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) e0060a8 HDDS-11196. Improve SCM WebUI Display (apache#6960) 22ddfb9 Revert "HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192)" 9f5bf43 HDDS-11457. Internal error on S3 CompleteMultipartUpload if parts are not specified (apache#7195) 10c47a1 HDDS-11459. Bump develocity-maven-extension to 1.22.1 (apache#7201) 50f2563 HDDS-11419. Fix waitForCheckpointDirectoryExist log message (apache#7199) a7d7e37 HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192) 5feb9ea HDDS-11453. OmSnapshotPurge should be in a different ozone manager double buffer batch (apache#7188) 703c4d5 HDDS-10984. Tool to restore SCM certificates from RocksDB. (apache#6781) d221065 HDDS-11440. Add a lastTransactionInfo field in SnapshotInfo to check for transactions in flight on the snapshot (apache#7179) e573701 HDDS-11448. Improve documentation in ContainerStateMachine (apache#7183) 0e49f7a HDDS-11449. Remove unnecessary log from client console. (apache#7184) cd251f2 HDDS-11438. Ensure DataInputBuffer is closed in OMPBHelper#convert (apache#7182) 4b47812 HDDS-11389. Incorrect number of deleted containers shown in Recon UI. (apache#7149) 0915f0b HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. (apache#7009) 0f16195 HDDS-11416. refactor ratis submit request avoid code duplicate (apache#7166) 86fe920 HDDS-11376. Improve ReplicationSupervisor to record replication metrics (apache#7140) CONFLICT: Merge conflict in hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandlerWithUnhealthyContainer.java MODIFIED: /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/DNContainerOperationClient.java /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java
aswinshakil
added a commit
to aswinshakil/ozone
that referenced
this pull request
Oct 2, 2024
COMMITS: 360fea5 HDDS-11494. Improve the duration option of freon ombg (apache#7246) 10d3b21 HDDS-11504. Update Ratis to 3.1.1. (apache#7257) ce46297 HDDS-11162. Improve Disk Usage page UI (apache#7214) c91f1c7 HDDS-11491. Avoid sharing clientId among deleting services (apache#7250) b0943d5 HDDS-11501. Improve logging in XceiverServerRatis (apache#7252) 55925ab HDDS-11502. Class path contains multiple SLF4J providers (apache#7255) d0ad836 HDDS-11472. Avoid recreating external access authorizer on OM state reload (apache#7238) 254db9e HDDS-11500. RootCARotationManager cancelling wrong task in notifyStatusChanged (apache#7251) 1e6e4b3 HDDS-11499. Remove redundant code from ECReconstructionCoordinator. (apache#7248) adb2821 HDDS-11490. Bump rollup to 3.29.5 (apache#7232) 189a9fe HDDS-11484. Validate javadoc in CI (apache#7245) 64a29c6 HDDS-11497. Bump commons-configuration2 to 2.11.0 (apache#7242) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snapshot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForPrefixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyInFailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snaps hot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForP refixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyIn FailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) e0060a8 HDDS-11196. Improve SCM WebUI Display (apache#6960) 22ddfb9 Revert "HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192)" 9f5bf43 HDDS-11457. Internal error on S3 CompleteMultipartUpload if parts are not specified (apache#7195) 10c47a1 HDDS-11459. Bump develocity-maven-extension to 1.22.1 (apache#7201) 50f2563 HDDS-11419. Fix waitForCheckpointDirectoryExist log message (apache#7199) a7d7e37 HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192) 5feb9ea HDDS-11453. OmSnapshotPurge should be in a different ozone manager double buffer batch (apache#7188) 703c4d5 HDDS-10984. Tool to restore SCM certificates from RocksDB. (apache#6781) d221065 HDDS-11440. Add a lastTransactionInfo field in SnapshotInfo to check for transactions in flight on the snapshot (apache#7179) e573701 HDDS-11448. Improve documentation in ContainerStateMachine (apache#7183) 0e49f7a HDDS-11449. Remove unnecessary log from client console. (apache#7184) cd251f2 HDDS-11438. Ensure DataInputBuffer is closed in OMPBHelper#convert (apache#7182) 4b47812 HDDS-11389. Incorrect number of deleted containers shown in Recon UI. (apache#7149) 0915f0b HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. (apache#7009) 0f16195 HDDS-11416. refactor ratis submit request avoid code duplicate (apache#7166) 86fe920 HDDS-11376. Improve ReplicationSupervisor to record replication metrics (apache#7140) CONFLICT: Merge conflict in hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandlerWithUnhealthyContainer.java MODIFIED: /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/DNContainerOperationClient.java /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java
aswinshakil
added a commit
to aswinshakil/ozone
that referenced
this pull request
Oct 2, 2024
COMMITS: 360fea5 HDDS-11494. Improve the duration option of freon ombg (apache#7246) 10d3b21 HDDS-11504. Update Ratis to 3.1.1. (apache#7257) ce46297 HDDS-11162. Improve Disk Usage page UI (apache#7214) c91f1c7 HDDS-11491. Avoid sharing clientId among deleting services (apache#7250) b0943d5 HDDS-11501. Improve logging in XceiverServerRatis (apache#7252) 55925ab HDDS-11502. Class path contains multiple SLF4J providers (apache#7255) d0ad836 HDDS-11472. Avoid recreating external access authorizer on OM state reload (apache#7238) 254db9e HDDS-11500. RootCARotationManager cancelling wrong task in notifyStatusChanged (apache#7251) 1e6e4b3 HDDS-11499. Remove redundant code from ECReconstructionCoordinator. (apache#7248) adb2821 HDDS-11490. Bump rollup to 3.29.5 (apache#7232) 189a9fe HDDS-11484. Validate javadoc in CI (apache#7245) 64a29c6 HDDS-11497. Bump commons-configuration2 to 2.11.0 (apache#7242) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snapshot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForPrefixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyInFailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) 95cfadd HDDS-11496. Bump maven-install-plugin to 3.1.3 (apache#7244) 0a999cf HDDS-11493. Bump sqlite-jdbc to 3.46.1.3 (apache#7243) a214a31 HDDS-11329. Update Ozone images to Rocky Linux-based runner (apache#7241) 56ddb85 HDDS-11371. Handle cases where OM does not have getServerDefaults() implemented. (apache#7130) b5097c7 HDDS-11347. Add rocks_tools_native lib check in Ozone CLI checknative subcommand (apache#7101) fb0bf77 HDDS-11489. Bump maven-site-plugin to 3.20.0 (apache#7226) 70e6e40 HDDS-11122. Fix javadoc warnings (apache#7234) acf3fdc HDDS-11458. Selective checks: trigger checkstyle for properties file changes (apache#7196) 6b87207 HDDS-11469. Statistics of Pipeline and Container (apache#7217) 1b8468b HDDS-11411. Snapshot garbage collection should not run when the keys are moved from a deleted snaps hot to the next snapshot in the chain (apache#7193) 1f86ce8 HDDS-10617. Unexpected number of files in ITestS3AContractGetFileStatusV1List (apache#7208) 73a3bcc HDDS-11467. Bump vite to 4.5.5 (apache#7212) d45aa1d HDDS-11460. Bump express to 4.21.0 (apache#7197) e2e30b8 HDDS-11354. Intermittent failure in TestOzoneManagerSnapshotAcl#testLookupKeyWithNotAllowedUserForP refixAcl (apache#7205) 0fcb645 HDDS-11477. [doc] Add configuration description for datanode docs (apache#7223) 3598ee3 HDDS-11464. Removed unused constants from OzoneConsts. (apache#7207) 8c0b54e HDDS-11408. Snapshot rename table entries are propagated incorrectly on snapshot deletes (apache#7200) 719bdf9 HDDS-11396. NPE due to empty Handler#clusterId (apache#7145) 40c4001 HDDS-10479. Add ozone admin ratis local raftMetaConf (apache#7170) 45f9138 HDDS-11394. Fix pipeline close --all command (apache#7138) 2b196d1 HDDS-11468. Enabled DB sync button (apache#7216) d3899d2 Clean up files created after TestKeyValueHandlerWithUnhealthyContainer#testMarkContainerUnhealthyIn FailedVolume (apache#7219) 70b8dd5 HDDS-11157. Improve Datanodes page UI (apache#7168) 151709a HDDS-11446. Downgrade picocli to 4.7.5 due to regression (apache#7215) 7a26aff HDDS-11158. Improve Pipelines page UI (apache#7171) c365aa0 HDDS-11181. Cleanup of unnecessary try-catch blocks (apache#7210) 88dd436 HDDS-11423. Implement equals operation for --filter option to ozone ldb scan (apache#7167) e0060a8 HDDS-11196. Improve SCM WebUI Display (apache#6960) 22ddfb9 Revert "HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192)" 9f5bf43 HDDS-11457. Internal error on S3 CompleteMultipartUpload if parts are not specified (apache#7195) 10c47a1 HDDS-11459. Bump develocity-maven-extension to 1.22.1 (apache#7201) 50f2563 HDDS-11419. Fix waitForCheckpointDirectoryExist log message (apache#7199) a7d7e37 HDDS-11456. Require successful dependency/licence checks for acceptance/compile/kubernetes (apache#7192) 5feb9ea HDDS-11453. OmSnapshotPurge should be in a different ozone manager double buffer batch (apache#7188) 703c4d5 HDDS-10984. Tool to restore SCM certificates from RocksDB. (apache#6781) d221065 HDDS-11440. Add a lastTransactionInfo field in SnapshotInfo to check for transactions in flight on the snapshot (apache#7179) e573701 HDDS-11448. Improve documentation in ContainerStateMachine (apache#7183) 0e49f7a HDDS-11449. Remove unnecessary log from client console. (apache#7184) cd251f2 HDDS-11438. Ensure DataInputBuffer is closed in OMPBHelper#convert (apache#7182) 4b47812 HDDS-11389. Incorrect number of deleted containers shown in Recon UI. (apache#7149) 0915f0b HDDS-10985. EC Reconstruction failed because the size of currentChunks was not equal to checksumBlockDataChunks. (apache#7009) 0f16195 HDDS-11416. refactor ratis submit request avoid code duplicate (apache#7166) 86fe920 HDDS-11376. Improve ReplicationSupervisor to record replication metrics (apache#7140) CONFLICT: Merge conflict in hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueHandlerWithUnhealthyContainer.java MODIFIED: /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/DNContainerOperationClient.java /Users/abalasubramanian/Documents/ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ReconcileContainerTask.java
sarvekshayr
pushed a commit
to sarvekshayr/ozone
that referenced
this pull request
Oct 7, 2024
…uble buffer batch (apache#7188)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Since purgeSnapshot transaction on a standalone basis is an idempotent operation. Once the directory gets deleted the previous transactions that have been performed on the snapshotted rocksdb would start failing on replay since those transactions have not been committed but the directory could have been partially deleted/ fully deleted. This could also lead to inconsistencies in the DB reads from the purged rocksdb if operations are not performed consciously.
Creating a separate rocksdb batch in ozone manager double buffer will ensure all transactions before have been flushed to the disk before purging the rocksdb, ensuring the previous ratis transactions will never get replayed.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-11453
How was this patch tested?
Changed existing unit tests.