Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Point Analysis #23931

Merged
merged 73 commits into from
Dec 22, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
6448677
ADd methods to Pytorch Benchmark to run Change Point analysis
AnandInguva Oct 26, 2022
b2239e7
Add GH issue API for Python
AnandInguva Oct 28, 2022
8d1dd98
Add support for commenting on the issue
AnandInguva Oct 31, 2022
a231516
Add a description about the failing test
AnandInguva Oct 31, 2022
d947923
Add gh_issue object for perf tests
AnandInguva Nov 1, 2022
3fca01c
Update API to read from parameters from config file
AnandInguva Nov 1, 2022
5b194ea
Add find sibling change point method
AnandInguva Nov 2, 2022
156af50
Add support for storing the metadata after regression issue is created
AnandInguva Nov 4, 2022
544b8fb
Add logic for finding sibling changepoint
AnandInguva Nov 4, 2022
5f3a94c
Update test_name
AnandInguva Nov 7, 2022
fc1de36
Update GH action
AnandInguva Nov 7, 2022
f825e4e
Update Description of the issue
AnandInguva Nov 7, 2022
c2cb8ae
Add commenting on the issue feature
AnandInguva Nov 7, 2022
13c24bb
Add logic for not creating alert when its the same changepoint
AnandInguva Nov 7, 2022
bde9e14
Update config file
AnandInguva Nov 8, 2022
7711303
Change owner name
AnandInguva Nov 8, 2022
fad21a2
fix lint, pydocs
AnandInguva Nov 8, 2022
26d72cc
Add two tests in the config file
AnandInguva Nov 9, 2022
bd39279
Fix lint
AnandInguva Nov 9, 2022
c4c4486
Add labels to the config file
AnandInguva Nov 9, 2022
18861e5
Fixup lints, and typehints
AnandInguva Nov 9, 2022
3fd0899
add additional params to the config file
AnandInguva Nov 9, 2022
0612c79
Refactor change point analysis code (#82)
AnandInguva Nov 16, 2022
af22414
Fixup lint
AnandInguva Nov 16, 2022
963827c
Changes based on comments
AnandInguva Nov 20, 2022
2600e24
Refactor fetch metrics from BQ
AnandInguva Nov 23, 2022
1ae5949
Add readme on perf alerting tool
AnandInguva Nov 23, 2022
0d7c72a
Add try catch statements
AnandInguva Nov 23, 2022
6d28eb5
Update github action
AnandInguva Nov 23, 2022
00bfee0
Add awaiting triage label
AnandInguva Nov 23, 2022
929124b
Add link to edvisive algorithm
AnandInguva Nov 23, 2022
a02f275
Check with previous change point timestamp in the sibling change poin…
AnandInguva Nov 29, 2022
801d79f
Changes based on comments on PR
AnandInguva Nov 29, 2022
af44c31
Fix test name
AnandInguva Nov 30, 2022
d167500
Merge remote-tracking branch 'upstream/master' into change-point-with…
AnandInguva Nov 30, 2022
2f49383
Refactor code
AnandInguva Dec 1, 2022
ef015b8
Move constants and helper functions to separate .py file
AnandInguva Dec 1, 2022
d96460f
Merge remote-tracking branch 'upstream/master' into change-point-with…
AnandInguva Dec 1, 2022
6db24e5
Refactoring
AnandInguva Dec 2, 2022
bdbf7a5
Rename files
AnandInguva Dec 2, 2022
d66b15d
Add tests
AnandInguva Dec 2, 2022
759d3c3
Refactor and add steps in the doc string
AnandInguva Dec 2, 2022
716a388
extend finding duplicate change points to search last 10 reported CPs
AnandInguva Dec 2, 2022
01c06e2
fix tests
AnandInguva Dec 2, 2022
8842b25
Fix whitespace lint
AnandInguva Dec 2, 2022
abef62d
Fix up lint
AnandInguva Dec 2, 2022
7104e61
Add _ to internal variables
AnandInguva Dec 5, 2022
fd57d19
Remove steps and add it to the PR description
AnandInguva Dec 5, 2022
e1b59c6
Make big_query_metrics_fetcher as a function
AnandInguva Dec 5, 2022
372ebc7
Add return in doc string
AnandInguva Dec 5, 2022
e8770cd
Add perf label and awaiting triage label default while creating alert
AnandInguva Dec 5, 2022
43185c9
Refactor
AnandInguva Dec 5, 2022
1797cc1
Modify loop
AnandInguva Dec 5, 2022
d130f1f
Fix up runtime errors
AnandInguva Dec 5, 2022
433f5fe
Refactor metrics fetcher
AnandInguva Dec 6, 2022
7672b99
Add test
AnandInguva Dec 6, 2022
b0f7b9c
Changes based on comments
AnandInguva Dec 12, 2022
327ea01
Remove the confusion on sorted order of metrics and timestamps
AnandInguva Dec 12, 2022
6fdfab4
Changes issue templates and add TODOs
AnandInguva Dec 12, 2022
117f5fe
Add triaging issues section
AnandInguva Dec 12, 2022
3a453cd
Fixup lint, whitespace
AnandInguva Dec 12, 2022
397e7c7
Add notebook
AnandInguva Dec 14, 2022
41a1a00
Changes based on comments
AnandInguva Dec 14, 2022
a0c3504
Mark optional params in readme
AnandInguva Dec 14, 2022
57a4ef0
correct triage link
AnandInguva Dec 17, 2022
fad6bbe
Add unit tests and fix lints
AnandInguva Dec 17, 2022
102d1b7
Fix up lint
AnandInguva Dec 19, 2022
6b83629
Apply suggestions from code review
AnandInguva Dec 22, 2022
4d7daf8
Change default optional param
AnandInguva Dec 22, 2022
6a22e77
modify mock tests
AnandInguva Dec 22, 2022
fed8018
add docstring
AnandInguva Dec 22, 2022
748ebd5
Change default values
AnandInguva Dec 22, 2022
c12b594
Add tests to GHA
AnandInguva Dec 22, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 53 additions & 2 deletions sdks/python/apache_beam/testing/analyzers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,61 @@ Sometimes, the change point found might be way back in time and could be irrelev
reported only when it was observed in the last 7 runs from the current run,
setting `num_runs_in_change_point_window=7` will achieve it.

## Register a test for performance alerts.
## Register a test for performance alerts

If a new test needs to be registered for the performance alerting tool, please add the required test parameters to the
config file.

[//]: # (TODO : Add triaging section)
## Triage performance alert issues

All the performance/load tests metrics defined at [beam/.test-infra/jenkins](https://github.com/apache/beam/tree/master/.test-infra/jenkins) are imported to [Grafana dashboards](http://104.154.241.245/d/1/getting-started?orgId=1) for visualization. Please
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there an url that is not based on IP address?

find the alerted test dashboard to find a spike in the metric values.

For example, for the below configuration,
* test: `apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks`
* metric_name: `mean_load_model_latency_milli_secs`

Grafana dashboard can be found at http://104.154.241.245/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7

If the dashboard for a test is not found, you can use the
code below to generate a plot for the given test, metric_name.

If the performance/load test store the results in BigQuery using this [schema](https://github.com/apache/beam/blob/83679216cce2d52dbeb7e837f06ca1d57b31d509/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py#L66),
then use the following code to fetch the metric_values for a `metric_name` for the last `30` runs and display a plot using matplotlib.

**NOTE**: Install matplotlib and pandas using `pip install matplotlib pandas`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, thanks. this would be better as a notebook in the repo, that people could refer to and update as necessary.

```
import pandas as pd
import matplotlib.pyplot as plt

from apache_beam.testing.load_tests import load_test_metrics_utils
from apache_beam.testing.load_tests.load_test_metrics_utils import BigQueryMetricsFetcher

bq_project = 'apache-beam-testing'
bq_dataset = '<bq-dataset-name>'
bq_table = '<bq-table>'
metric_name = '<perf-alerted-metric-name>'

query = f"""
SELECT *
FROM {bq_project}.{bq_dataset}.{bq_table}
WHERE CONTAINS_SUBSTR(({load_test_metrics_utils.METRICS_TYPE_LABEL}), '{metric_name}')
ORDER BY {load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL} DESC
LIMIT 30
"""

big_query_metrics_fetcher = BigQueryMetricsFetcher()
metric_data: pd.DataFrame = big_query_metrics_fetcher.fetch(query=query)
# sort the data to view it in chronological order.
metric_data.sort_values(
by=[load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL], inplace=True)

metric_data.plot(x=load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL,
y=load_test_metrics_utils.VALUE_LABEL)
plt.show()
```

If you confirm there is a change in the pattern of the values for a test, find the timestamp of when that change happened
and use that timestamp to find possible culprit commit.

If the performance alert is a `false positive`, close the issue as `Close as not planned`.
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,11 @@
# TODO: Provide a better debugging tool for the user to visualize metrics.
# For example, a link to dashboards or query to get recent data to analyze,etc.
AnandInguva marked this conversation as resolved.
Show resolved Hide resolved
_ISSUE_DESCRIPTION_TEMPLATE = """
Performance change found in the test: `{}` for the metric: `{}`.
Performance change found in the
test: `{}` for the metric: `{}`.

For more information on how to triage the alerts, please look at
`Triage performance alert issues` section of the README.
"""
_METRIC_INFO_TEMPLATE = "timestamp: {}, metric_value: `{}`"
_AWAITING_TRIAGE_LABEL = 'awaiting triage'
Expand Down
4 changes: 2 additions & 2 deletions sdks/python/apache_beam/testing/analyzers/tests_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#

test_1:
test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_16
test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
AnandInguva marked this conversation as resolved.
Show resolved Hide resolved
metrics_dataset: beam_run_inference
metrics_table: torch_inference_imagenet_results_resnet152
project: apache-beam-testing
Expand All @@ -25,7 +25,7 @@ test_1:
- run-inference
# TODO: Remove before merging.
min_runs_between_change_points: 5
AnandInguva marked this conversation as resolved.
Show resolved Hide resolved
num_runs_in_change_point_window: 14
num_runs_in_change_point_window: 30

test_2:
test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
Expand Down