Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change max query size for GetMetricData API to 500 and add RecentlyActive config #33105

Merged
merged 17 commits into from
Oct 11, 2022
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff]
- in module/windows/perfmon, changed collection method of the second counter value required to create a displayable value {pull}32305[32305]
- Fix and improve AWS metric period calculation to avoid zero-length intervals {pull}32724[32724]
- Add missing cluster metadata to k8s module metricsets {pull}32979[32979] {pull}33032[33032]
- Change max query size for GetMetricData API to 500 {pull}33105[33105]

*Packetbeat*

Expand Down
37 changes: 20 additions & 17 deletions x-pack/metricbeat/module/aws/utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,12 @@ func GetListMetricsOutput(namespace string, regionName string, svcCloudwatch clo

// GetMetricDataResults function uses MetricDataQueries to get metric data output.
func GetMetricDataResults(metricDataQueries []types.MetricDataQuery, svc cloudwatch.GetMetricDataAPIClient, startTime time.Time, endTime time.Time) ([]types.MetricDataResult, error) {
maxQuerySize := 100
maxQuerySize := 500
kaiyan-sheng marked this conversation as resolved.
Show resolved Hide resolved
getMetricDataOutput := &cloudwatch.GetMetricDataOutput{NextToken: nil}

// Split metricDataQueries into smaller slices that length no longer than 100.
// 100 is defined in maxQuerySize.
// To avoid ValidationError: The collection MetricDataQueries must not have a size greater than 100.
// Split metricDataQueries into smaller slices that length no longer than 500.
// 500 is defined in maxQuerySize.
// To avoid ValidationError: The collection MetricDataQueries must not have a size greater than 500.
for i := 0; i < len(metricDataQueries); i += maxQuerySize {
metricDataQueriesPartial := metricDataQueries[i:int(math.Min(float64(i+maxQuerySize), float64(len(metricDataQueries))))]
if len(metricDataQueriesPartial) == 0 {
Expand Down Expand Up @@ -111,19 +111,22 @@ func CheckTimestampInArray(timestamp time.Time, timestampArray []time.Time) (boo

// FindTimestamp function checks MetricDataResults and find the timestamp to collect metrics from.
// For example, MetricDataResults might look like:
// metricDataResults = [{
// Id: "sqs0",
// Label: "testName SentMessageSize",
// StatusCode: Complete,
// Timestamps: [2019-03-11 17:45:00 +0000 UTC],
// Values: [981]
// } {
// Id: "sqs1",
// Label: "testName NumberOfMessagesSent",
// StatusCode: Complete,
// Timestamps: [2019-03-11 17:45:00 +0000 UTC,2019-03-11 17:40:00 +0000 UTC],
// Values: [0.5,0]
// }]
//
// metricDataResults = [{
// Id: "sqs0",
// Label: "testName SentMessageSize",
// StatusCode: Complete,
// Timestamps: [2019-03-11 17:45:00 +0000 UTC],
// Values: [981]
// } {
//
// Id: "sqs1",
// Label: "testName NumberOfMessagesSent",
// StatusCode: Complete,
// Timestamps: [2019-03-11 17:45:00 +0000 UTC,2019-03-11 17:40:00 +0000 UTC],
// Values: [0.5,0]
// }]
//
// This case, we are collecting values for both metrics from timestamp 2019-03-11 17:45:00 +0000 UTC.
func FindTimestamp(getMetricDataResults []types.MetricDataResult) time.Time {
timestamp := time.Time{}
Expand Down