feat: add retry mechanism in read blob metadata

[Desiki-high]

Relevant Issue (if applicable)

#1613

Details

When reading blob size from blob metadata, we should retry reading from the remote if an error occurs.

• Handle the BackendError.
• Set the max retry times to 3.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

• I have updated the documentation accordingly.
• I have added tests to cover my changes.

[codecov]

Codecov Report

Attention: Patch coverage is 27.27273% with 16 lines in your changes missing coverage. Please review.

Project coverage is 61.27%. Comparing base (7fc9ede) to head (e04b5ca).
Report is 2 commits behind head on master.

Files with missing lines	Patch %	Lines
storage/src/meta/mod.rs	27.27%	15 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1617      +/-   ##
==========================================
- Coverage   61.29%   61.27%   -0.02%     
==========================================
  Files         146      146              
  Lines       48143    48156      +13     
  Branches    46110    46123      +13     
==========================================
+ Hits        29507    29509       +2     
- Misses      17075    17085      +10     
- Partials     1561     1562       +1     

Files with missing lines	Coverage Δ
storage/src/meta/mod.rs	72.21% <27.27%> (-0.31%)	⬇️

... and 3 files with indirect coverage changes

[imeoer]

LGTM!