[Enhancement] Change CatIndexTool implementation from _cat/index action to _list/index action

[zane-neo]

Is your feature request related to a problem?

As _cat/index API is not performant in titan cluster and a new paginated _list/index API has been introduced recently: opensearch-project/OpenSearch#14258. We need to review the implementation of CatIndexTool as it uses the old RestIndicesAction which could cause issue on titan clusters. In this PR a new cluster setting cat.indices.response.limit.number_of_indices is introduced to control the maximum indices number in _cat/index API, so CatIndexTool could throw exception in a titan cluster w/o modifying the implementation.

What solution would you like?
Replace the CatIndexTool implementation to _list/index approach.

What alternatives have you considered?
Still use _cat/index API and honer the cluster setting cat.indices.response.limit.number_of_indices to make sure no exception thrown during runtime, but the drawback is the tool may not able to fetch all the indices in a titan cluster since the amount of the indices might more than the setting's value.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

[dhrubo-os]

What do you mean by titan cluster?

[ylwu-amzn]

Why not build another ListIndexTool?
List index API introduced in 2.18 https://opensearch.org/docs/latest/api-reference/list/list-indices/. Seems hard to backport fix/enhance of CatIndexTool to older versions by changing it to _list/indices ,

[zane-neo]

What do you mean by titan cluster?

Titan cluster means clusters that has a lot of number of nodes and on those clusters a _cat/index invocation might never return.

[zane-neo]

Why not build another ListIndexTool? List index API introduced in 2.18 https://opensearch.org/docs/latest/api-reference/list/list-indices/. Seems hard to backport fix/enhance of CatIndexTool to older versions by changing it to _list/indices ,

Agreed, I'll add a new tool instead of modifying the CatIndexTool.