Add support of reading from archives in S3 #62259

divanik · 2024-04-03T16:58:19Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Earlier our s3 storage and s3 table function didn't support selecting from archive files. I created a solution that allows to iterate over files inside archives in S3.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

Modify your CI run:

NOTE: If your merge the PR with modified CI you MUST KNOW what you are doing
NOTE: Checked options will be applied if set before CI RunConfig/PrepareRunConfig step

Include tests (required builds will be added automatically):

Exclude tests:

Extra options:

do not test (only style check)
disable merge-commit (no merge from master before tests)
disable CI cache (job reuse)

Only specified batches in multi-batch jobs:

1
2
3
4

CLAassistant · 2024-04-03T16:58:27Z

All committers have signed the CLA.

robot-ch-test-poll1 · 2024-04-03T17:00:16Z

This is an automated comment for commit 851a849 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
A Sync	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	⏳ pending
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	⏳ pending
Mergeable Check	Checks if all other necessary checks are successful	❌ failure
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure

Successful checks

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	✅ success
ClickBench	Runs [ClickBench](https://github.com/ClickHouse/ClickBench/) with instant-attach table	✅ success
ClickHouse build check	Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docs check	Builds and tests the documentation	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integrational tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	✅ success
PR Check	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	✅ success

divanik · 2024-04-09T12:03:44Z

SELECT _file, count() FROM s3('https://s3-us-west-1.amazonaws.com/umbrella-static/top-1m-2018-01-1*.csv.zip :: *.csv', 'NOSIGN') GROUP BY _file ORDER BY _file;

file	count()
top-1m-2018-01-11.csv.zip	1000000
top-1m-2018-01-12.csv.zip	1000000
top-1m-2018-01-13.csv.zip	1000000
top-1m-2018-01-14.csv.zip	1000000
top-1m-2018-01-15.csv.zip	1000000
top-1m-2018-01-16.csv.zip	1000000
top-1m-2018-01-17.csv.zip	1000000
top-1m-2018-01-18.csv.zip	1000000
top-1m-2018-01-19.csv.zip	1000000
top-1m-2018-01-20.csv.zip	1000000

src/Storages/StorageS3.cpp

tests/queries/0_stateless/03036_reading_s3_archives.sql

tests/queries/0_stateless/03036_reading_s3_archives.reference

StashOfCode · 2024-04-14T23:20:44Z

Hey! This is an amazing feature. Which Clickhouse version is it gonna be in?

alexey-milovidov · 2024-04-23T01:03:32Z

@StashOfCode 24.4 most likely.

src/IO/S3/URI.cpp

src/Storages/StorageS3.h

src/IO/S3/URI.h

src/IO/S3/URI.cpp

src/Storages/StorageS3.cpp

tests/queries/0_stateless/03036_reading_s3_archives.sql

src/Storages/StorageS3.cpp

src/Storages/StorageS3.h

src/Storages/StorageS3.cpp

kssenii · 2024-04-26T15:47:54Z

src/Storages/StorageS3.cpp

@@ -1794,7 +2047,8 @@ namespace
                || getContext()->getSettingsRef().schema_inference_mode != SchemaInferenceMode::UNION)
                return;

-            String source = fs::path(configuration.url.uri.getHost() + std::to_string(configuration.url.uri.getPort())) / configuration.url.bucket / current_key_with_info->key;
+            String source = fs::path(configuration.url.uri.getHost() + std::to_string(configuration.url.uri.getPort()))
+                / configuration.url.bucket / current_key_with_info->getPath();


getPath() returns bucket + "/" + key_with_info->key (as an example, for non-archive case), so here as a result of this change we will at bucket twice.

src/Storages/StorageS3.cpp

alexey-milovidov · 2024-05-02T19:11:28Z

Don't forget to squash:

Kek
Kek
Kek

src/Storages/StorageS3.h

tests/queries/0_stateless/03036_reading_s3_archives.sql

src/Storages/StorageS3.h

src/Storages/StorageS3.cpp

tests/queries/0_stateless/03036_schema_inference_cache_s3_archives.sql

tavplubix · 2024-05-16T15:27:42Z

@divanik, test_storage_s3/test.py::test_seekable_formats_url became flaky after this PR: CI DB

tavplubix · 2024-05-16T15:28:10Z

Ah, sorry, not after this one...

divanik · 2024-05-31T14:20:31Z

Contination: #64703

divanik self-assigned this Apr 3, 2024

robot-ch-test-poll1 added the pr-feature Pull request with new product feature label Apr 3, 2024

divanik changed the title ~~divanik/fix archieve support in s3~~ Add support of archives in S3 Apr 3, 2024

kssenii self-assigned this Apr 3, 2024

nikitamikhaylov unassigned divanik Apr 3, 2024

divanik changed the title ~~Add support of archives in S3~~ Add archives support to S3 Apr 4, 2024

alesapin marked this pull request as ready for review April 8, 2024 18:23

alesapin marked this pull request as draft April 8, 2024 18:23

alesapin reviewed Apr 9, 2024

View reviewed changes

src/Storages/StorageS3.cpp Outdated Show resolved Hide resolved

src/Storages/StorageS3.cpp Outdated Show resolved Hide resolved

divanik marked this pull request as ready for review April 9, 2024 14:11

antonio2368 reviewed Apr 10, 2024

View reviewed changes

tests/queries/0_stateless/03036_reading_s3_archives.sql Show resolved Hide resolved

antonio2368 reviewed Apr 11, 2024

View reviewed changes

tests/queries/0_stateless/03036_reading_s3_archives.reference Outdated Show resolved Hide resolved

divanik mentioned this pull request Apr 11, 2024

Direct import from archives don't work for table function s3 #54726

Closed

divanik linked an issue Apr 11, 2024 that may be closed by this pull request

Direct import from archives don't work for table function s3 #54726

Closed

divanik requested review from alesapin and antonio2368 April 11, 2024 23:10

divanik requested a review from kssenii April 22, 2024 14:30

kssenii reviewed Apr 23, 2024

View reviewed changes

divanik requested a review from kssenii April 26, 2024 15:01

kssenii reviewed Apr 26, 2024

View reviewed changes

divanik changed the title ~~Add archives support to S3~~ Add support of reading from archives in S3 Apr 30, 2024

robot-ch-test-poll1 added the submodule changed At least one submodule changed in this PR. label May 2, 2024

divanik removed the submodule changed At least one submodule changed in this PR. label May 2, 2024

divanik force-pushed the divanik/fix_archieve_support_in_s3 branch from 081688a to 62c27b9 Compare May 8, 2024 11:55

kssenii reviewed May 13, 2024

View reviewed changes

divanik force-pushed the divanik/fix_archieve_support_in_s3 branch 7 times, most recently from 8c08471 to af76003 Compare May 13, 2024 11:51

Add archives reading support to s3

d468a0a

divanik force-pushed the divanik/fix_archieve_support_in_s3 branch from af76003 to d468a0a Compare May 13, 2024 12:01

divanik added 3 commits May 13, 2024 12:04

Resolve conflicts

faef880

Resolve several issues

dc8f8a9

Add schema inference test

6ec868a

kssenii approved these changes May 13, 2024

View reviewed changes

divanik added 2 commits May 14, 2024 09:25

Add no fasttest tag

1eb72e7

Possible fix of stability of schema_infernce_cache test

3737e6f

kssenii reviewed May 14, 2024

View reviewed changes

tests/queries/0_stateless/03036_schema_inference_cache_s3_archives.sql Outdated Show resolved Hide resolved

Fix test

851a849

divanik enabled auto-merge May 15, 2024 09:20

divanik disabled auto-merge May 15, 2024 09:20

divanik added this pull request to the merge queue May 16, 2024

Merged via the queue into ClickHouse:master with commit 9b09550 May 16, 2024
228 of 231 checks passed

divanik deleted the divanik/fix_archieve_support_in_s3 branch May 16, 2024 10:59

robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label May 16, 2024

azat mentioned this pull request May 18, 2024

tests: fix expected error for 03036_reading_s3_archives (fixes CI) #64089

Merged

divanik mentioned this pull request May 31, 2024

Further improvements for working with archives in object storages #64703

Open

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of reading from archives in S3 #62259

Add support of reading from archives in S3 #62259

divanik commented Apr 3, 2024 •

edited

Loading

CLAassistant commented Apr 3, 2024 •

edited

Loading

robot-ch-test-poll1 commented Apr 3, 2024 •

edited by robot-ch-test-poll2

Loading

divanik commented Apr 9, 2024

StashOfCode commented Apr 14, 2024 •

edited

Loading

alexey-milovidov commented Apr 23, 2024

kssenii Apr 26, 2024

alexey-milovidov commented May 2, 2024

tavplubix commented May 16, 2024

tavplubix commented May 16, 2024

divanik commented May 31, 2024

Add support of reading from archives in S3 #62259

Add support of reading from archives in S3 #62259

Conversation

divanik commented Apr 3, 2024 • edited Loading

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Modify your CI run:

Include tests (required builds will be added automatically):

Exclude tests:

Extra options:

Only specified batches in multi-batch jobs:

CLAassistant commented Apr 3, 2024 • edited Loading

robot-ch-test-poll1 commented Apr 3, 2024 • edited by robot-ch-test-poll2 Loading

divanik commented Apr 9, 2024

StashOfCode commented Apr 14, 2024 • edited Loading

alexey-milovidov commented Apr 23, 2024

kssenii Apr 26, 2024

Choose a reason for hiding this comment

alexey-milovidov commented May 2, 2024

tavplubix commented May 16, 2024

tavplubix commented May 16, 2024

divanik commented May 31, 2024

divanik commented Apr 3, 2024 •

edited

Loading

CLAassistant commented Apr 3, 2024 •

edited

Loading

robot-ch-test-poll1 commented Apr 3, 2024 •

edited by robot-ch-test-poll2

Loading

StashOfCode commented Apr 14, 2024 •

edited

Loading