Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opendata Refactor #434

Merged
merged 36 commits into from
Oct 18, 2024
Merged

Opendata Refactor #434

merged 36 commits into from
Oct 18, 2024

Conversation

scezen
Copy link
Collaborator

@scezen scezen commented Sep 13, 2024

Objective

Improve Opendata pages and services by splitting the data between GTIN and ISBN datasets with enhanced UI.

Summary of Changes

  • OpenDataController:

    • Added new routes for GTIN and ISBN datasets (/opendata/gtin, /opendata/isbn).
    • Enhanced model with additional attributes (e.g., gtinLastUpdated, isbnFileSize, etc.).
    • Implemented getMultipleExposedUrls() method to expose multiple sitemap entries.

  • OpenDataService:

    • Added generation and management of both GTIN and ISBN datasets.
    • Implemented processAndCreateZip() to handle dataset compression.
    • Implemented HealthIndicator to monitor the availability of GTIN and ISBN ZIP files.
    • New OpenDataConfig conf file (download speed and concurrent download limits)

  • Frontend Pages:

    • Updated main Opendata page to display separate GTIN and ISBN datasets.
    • Added new detailed pages for GTIN and ISBN downloads, with individual file size and last updated information.

scezen and others added 30 commits October 17, 2024 18:56
@scezen
Separate processing of ISBN and GTIN datasets in opendata export
dec7bd7
@scezen
Implement processAndAddToZip method
f4582ca
@scezen
Typo
3a692d4
@scezen
Small fixes, comments
ad86e4e
@scezen
Re-enable scheduling
5d7820b
@scezen
Separate directory preparation from generateOpendata
37dbd8d
@scezen
Extract file moving logic to separate method
c789a53
@scezen
Use try-with-resources for automatic resource management
4fa529a
@scezen
Update OpenDataController + UiConfig
2094080
@scezen
Add missing cacheable methods for ISBN and GTIN file info
849bdf6
@scezen
Update OpenDataController to handle multiple file types and improve c…
35da3f3 
…larity
@scezen
Fix ZIP entry handling in processAndCreateZip
8e9b624
@scezen
Add GTIN and ISBN specific pages + Controller Update
395ffac
@scezen
Notes @scezen
d342884
@scezen
Typo
b4a1d60
@scezen
Adding OpenDataConfig
542995c
@scezen
Frontend Tweaks + ISBN Headers
bb9e218
@scezen
Fix header logic and filtering for ISBN and GTIN datasets
51845cf
@scezen
Typo
c8cb7e6
@scezen
Removing scheduled props
ca6d168
@scezen
ProductRepository
1102c0a
@scezen
Working session
d2d6cd1
@scezen
Specific ISBN attributes and null values handling + Refactor and cleanup
65202c2
@scezen
Cleanup
36363c3
@scezen
Add health check and timing metrics
30e8f8c
@scezen
Separate processing of ISBN and GTIN datasets in opendata export
42f4553
@scezen
Implement processAndAddToZip method
c3c2c8e
@scezen
Typo
37b8b22
@scezen
Extract file moving logic to separate method
6dfe0ad
@scezen
Working session
505400c
@GoulvenF
Copy link
Contributor

Thanks @scezen !

@GoulvenF GoulvenF merged commit 3fc196b into main Oct 18, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@scezen @GoulvenF