[Bug]: ElasticsearchIO connector does not properly estimate index size

[JeffBolle]

In the process of investigating the issue reported here:
https://stackoverflow.com/questions/74390325/how-to-enable-elasticsearchio-parallel-reads-in-apache-beam

it appears that the method used by the ElasticsearchIO connector to get the estimated size of the data in the response is not accounting for the case where the configured index is an alias or a datastream or an index pattern which can point to multiple indexes.

The original issue was a query that returns over 100 million documents for processing in the pipeline was unable to scale and was only processing at a rate of 40 / second.

As discussed in the stackoverflow thread, the code here:

beam/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java

Line 871 in c7f2cab

public long getEstimatedSizeBytes(PipelineOptions options) throws IOException {

is not properly accounting for a number of scenarios where the index name returned by ElasticSearch is different than connectionConfiguration.getIndex().

ElasticSearch should be relied upon to return the proper indexes for a given stats query, and as such the _all object should be used instead of the indicies top level object. If there are other cases where the _all object isn't appropriate, then the code should iterate through all of the indicies returned under the indices field and sum the total store size, and not simply try to match the configured index.

Issue Priority

Priority: 2

Issue Component

Component: io-java-elasticsearch

[robertwb]

At the very least we should provide a way for the end user to specify the size or degree of splitting until this is resolved.

[aromanenko-dev]

CC: @egalpin

[JeffBolle]

I'm working on a PR for this, but its a trivial change. And since I'm not a contributor it might be faster for someone else to actually run this through the process.
I've attached the patch if anyone wants to have a look.
0001-Read-from-_all-instead-of-_indices-for-index-stats.patch.txt

[egalpin]

Thanks @JeffBolle for raising this and for proposing the patch. I can have a look at the patch, but definitely would not want to rob you of the chance to contribute your change directly if you're interested in doing so!

[egalpin]

Ok I think the patch is close but not quite right. using _all would take the size of all indices in the cluster, but that may not be applicable to the size of data that one will be reading. Instead, if we use the stats API at the index level (GET /my-index-name/_stats) and then use _all, it contains data for all aliases of my-index-name.

I believe this would fulfill the original goal. Thoughts @JeffBolle ?

My apologies, this is exactly what was already proposed in the patch. I'll read more slowly next time haha

[JeffBolle]

Thanks @JeffBolle for raising this and for proposing the patch. I can have a look at the patch, but definitely would not want to rob you of the chance to contribute your change directly if you're interested in doing so!

I appreciate that, but for now I think this is the best path. Better to get the issue resolved quickly. If the next one involves more work to address properly I might want credit for that.