Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Point Analysis #23931

Merged
merged 73 commits into from
Dec 22, 2022
Merged
Show file tree
Hide file tree
Changes from 63 commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
6448677
ADd methods to Pytorch Benchmark to run Change Point analysis
AnandInguva Oct 26, 2022
b2239e7
Add GH issue API for Python
AnandInguva Oct 28, 2022
8d1dd98
Add support for commenting on the issue
AnandInguva Oct 31, 2022
a231516
Add a description about the failing test
AnandInguva Oct 31, 2022
d947923
Add gh_issue object for perf tests
AnandInguva Nov 1, 2022
3fca01c
Update API to read from parameters from config file
AnandInguva Nov 1, 2022
5b194ea
Add find sibling change point method
AnandInguva Nov 2, 2022
156af50
Add support for storing the metadata after regression issue is created
AnandInguva Nov 4, 2022
544b8fb
Add logic for finding sibling changepoint
AnandInguva Nov 4, 2022
5f3a94c
Update test_name
AnandInguva Nov 7, 2022
fc1de36
Update GH action
AnandInguva Nov 7, 2022
f825e4e
Update Description of the issue
AnandInguva Nov 7, 2022
c2cb8ae
Add commenting on the issue feature
AnandInguva Nov 7, 2022
13c24bb
Add logic for not creating alert when its the same changepoint
AnandInguva Nov 7, 2022
bde9e14
Update config file
AnandInguva Nov 8, 2022
7711303
Change owner name
AnandInguva Nov 8, 2022
fad21a2
fix lint, pydocs
AnandInguva Nov 8, 2022
26d72cc
Add two tests in the config file
AnandInguva Nov 9, 2022
bd39279
Fix lint
AnandInguva Nov 9, 2022
c4c4486
Add labels to the config file
AnandInguva Nov 9, 2022
18861e5
Fixup lints, and typehints
AnandInguva Nov 9, 2022
3fd0899
add additional params to the config file
AnandInguva Nov 9, 2022
0612c79
Refactor change point analysis code (#82)
AnandInguva Nov 16, 2022
af22414
Fixup lint
AnandInguva Nov 16, 2022
963827c
Changes based on comments
AnandInguva Nov 20, 2022
2600e24
Refactor fetch metrics from BQ
AnandInguva Nov 23, 2022
1ae5949
Add readme on perf alerting tool
AnandInguva Nov 23, 2022
0d7c72a
Add try catch statements
AnandInguva Nov 23, 2022
6d28eb5
Update github action
AnandInguva Nov 23, 2022
00bfee0
Add awaiting triage label
AnandInguva Nov 23, 2022
929124b
Add link to edvisive algorithm
AnandInguva Nov 23, 2022
a02f275
Check with previous change point timestamp in the sibling change poin…
AnandInguva Nov 29, 2022
801d79f
Changes based on comments on PR
AnandInguva Nov 29, 2022
af44c31
Fix test name
AnandInguva Nov 30, 2022
d167500
Merge remote-tracking branch 'upstream/master' into change-point-with…
AnandInguva Nov 30, 2022
2f49383
Refactor code
AnandInguva Dec 1, 2022
ef015b8
Move constants and helper functions to separate .py file
AnandInguva Dec 1, 2022
d96460f
Merge remote-tracking branch 'upstream/master' into change-point-with…
AnandInguva Dec 1, 2022
6db24e5
Refactoring
AnandInguva Dec 2, 2022
bdbf7a5
Rename files
AnandInguva Dec 2, 2022
d66b15d
Add tests
AnandInguva Dec 2, 2022
759d3c3
Refactor and add steps in the doc string
AnandInguva Dec 2, 2022
716a388
extend finding duplicate change points to search last 10 reported CPs
AnandInguva Dec 2, 2022
01c06e2
fix tests
AnandInguva Dec 2, 2022
8842b25
Fix whitespace lint
AnandInguva Dec 2, 2022
abef62d
Fix up lint
AnandInguva Dec 2, 2022
7104e61
Add _ to internal variables
AnandInguva Dec 5, 2022
fd57d19
Remove steps and add it to the PR description
AnandInguva Dec 5, 2022
e1b59c6
Make big_query_metrics_fetcher as a function
AnandInguva Dec 5, 2022
372ebc7
Add return in doc string
AnandInguva Dec 5, 2022
e8770cd
Add perf label and awaiting triage label default while creating alert
AnandInguva Dec 5, 2022
43185c9
Refactor
AnandInguva Dec 5, 2022
1797cc1
Modify loop
AnandInguva Dec 5, 2022
d130f1f
Fix up runtime errors
AnandInguva Dec 5, 2022
433f5fe
Refactor metrics fetcher
AnandInguva Dec 6, 2022
7672b99
Add test
AnandInguva Dec 6, 2022
b0f7b9c
Changes based on comments
AnandInguva Dec 12, 2022
327ea01
Remove the confusion on sorted order of metrics and timestamps
AnandInguva Dec 12, 2022
6fdfab4
Changes issue templates and add TODOs
AnandInguva Dec 12, 2022
117f5fe
Add triaging issues section
AnandInguva Dec 12, 2022
3a453cd
Fixup lint, whitespace
AnandInguva Dec 12, 2022
397e7c7
Add notebook
AnandInguva Dec 14, 2022
41a1a00
Changes based on comments
AnandInguva Dec 14, 2022
a0c3504
Mark optional params in readme
AnandInguva Dec 14, 2022
57a4ef0
correct triage link
AnandInguva Dec 17, 2022
fad6bbe
Add unit tests and fix lints
AnandInguva Dec 17, 2022
102d1b7
Fix up lint
AnandInguva Dec 19, 2022
6b83629
Apply suggestions from code review
AnandInguva Dec 22, 2022
4d7daf8
Change default optional param
AnandInguva Dec 22, 2022
6a22e77
modify mock tests
AnandInguva Dec 22, 2022
fed8018
add docstring
AnandInguva Dec 22, 2022
748ebd5
Change default values
AnandInguva Dec 22, 2022
c12b594
Add tests to GHA
AnandInguva Dec 22, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .github/workflows/run_perf_alert_tool.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# To learn more about GitHub Actions in Apache Beam check the CI.md

name: Run performance alerting tool on Python load tests.

on:
schedule:
- cron: '5 * * * *' # TODO: Change the window of the cron job.

jobs:
python_run_change_point_analysis:
name: Run Change Point Analysis.
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Install python
uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Get Apache Beam Build dependencies
working-directory: ./sdks/python
run: pip install pip setuptools --upgrade && pip install -r build-requirements.txt
- name: Install Apache Beam
working-directory: ./sdks/python
run: pip install -e .[gcp]
- name: Install signal-processing-algorithms
run: pip install signal-processing-algorithms
- name: Install Pandas, yaml, requests
run: pip install pandas PyYAML requests
- name: Run Change Point Analysis.
working-directory: ./sdks/python/apache_beam/testing/analyzers
shell: bash
run: python analysis.py
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
97 changes: 97 additions & 0 deletions sdks/python/apache_beam/testing/analyzers/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Performance alerts for Beam Python performance and load tests

## Alerts
Performance regressions or improvements detected with the [Change Point Analysis](https://en.wikipedia.org/wiki/Change_detection) using [edivisive](https://github.com/apache/beam/blob/0a91d139dea4276dc46176c4cdcdfce210fc50c4/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L30)
analyzer are automatically filed as Beam GitHub issues with a label `perf-alert`.

The GitHub issue description will contain the information on the affected test and metric by providing the metric values for N consecutive runs with timestamps
before and after the observed change point. Observed change point is pointed as `Anomaly` in the issue description.

Example: [sample perf alert GitHub issue](https://github.com/AnandInguva/beam/issues/83).

If a performance alert is created on a test, a GitHub issue will be created and the GitHub issue metadata such as GitHub issue
URL, issue number along with the change point value and timestamp are exported to BigQuery. This data will be used to analyze the next change point observed on the same test to
update already created GitHub issue or ignore performance alert by not creating GitHub issue to avoid duplicate issue creation.

## Config file structure
The config file defines the structure to run change point analysis on a given test. To add a test to the config file,
please follow the below structure.

**NOTE**: The Change point analysis only supports reading the metric data from Big Query for now.

```
# the test_1 must be a unique id.
tvalentyn marked this conversation as resolved.
Show resolved Hide resolved
test_1:
test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
source: big_query
metrics_dataset: beam_run_inference
metrics_table: torch_inference_imagenet_results_resnet152
project: apache-beam-testing
metric_name: mean_load_model_latency_milli_secs
labels:
- run-inference
min_runs_between_change_points: 5
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think reasonable default values should obviate the need to configure some of the parameters.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I set the default values in the .py file. But if someone provides values in the config file for a particular test, then the default values will be overridden

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's mark which are optional.

num_runs_in_change_point_window: 7
```

**NOTE**: `test_name` should be in the format `apache_beam.foo.bar`. It should point to a single test target.

**Note**: If the source is **BigQuery**, the metrics_dataset, metrics_table, project and metric_name should match with the values defined for performance/load tests.
The above example uses this [test configuration](https://github.com/apache/beam/blob/0a91d139dea4276dc46176c4cdcdfce210fc50c4/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L30)
to fill up the values required to fetch the data from source.

### Different ways to avoid false positive change points

**min_runs_between_change_points**: As the metric data moves across the runs, the change point analysis can place the
change point in a slightly different place. These change points refer to the same regression and are just noise.
When we find a new change point, we will search up to the `min_runs_between_change_points` in both directions from the
current change point. If an existing change point is found within the distance, then the current change point will be
suppressed.

**num_runs_in_change_point_window**: This defines how many runs to consider from the most recent run to be in change point window.
Sometimes, the change point found might be way back in time and could be irrelevant. For a test, if a change point needs to be
reported only when it was observed in the last 7 runs from the current run,
setting `num_runs_in_change_point_window=7` will achieve it.

## Register a test for performance alerts

If a new test needs to be registered for the performance alerting tool, please add the required test parameters to the
config file.

## Triage performance alert issues

All the performance/load tests metrics defined at [beam/.test-infra/jenkins](https://github.com/apache/beam/tree/master/.test-infra/jenkins) are imported to [Grafana dashboards](http://104.154.241.245/d/1/getting-started?orgId=1) for visualization. Please
find the alerted test dashboard to find a spike in the metric values.

For example, for the below configuration,
* test: `apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks`
* metric_name: `mean_load_model_latency_milli_secs`

Grafana dashboard can be found at http://104.154.241.245/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7

If the dashboard for a test is not found, you can use the
notebook `analyze_metric_data.ipynb` to generate a plot for the given test, metric_name.

If you confirm there is a change in the pattern of the values for a test, find the timestamp of when that change happened
and use that timestamp to find possible culprit commit.

If the performance alert is a `false positive`, close the issue as `Close as not planned`.
16 changes: 16 additions & 0 deletions sdks/python/apache_beam/testing/analyzers/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
172 changes: 172 additions & 0 deletions sdks/python/apache_beam/testing/analyzers/analyze_metric_data.ipynb
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "code",
"source": [
"# Licensed to the Apache Software Foundation (ASF) under one\n",
"# or more contributor license agreements. See the NOTICE file\n",
"# distributed with this work for additional information\n",
"# regarding copyright ownership. The ASF licenses this file\n",
"# to you under the Apache License, Version 2.0 (the\n",
"# \"License\"); you may not use this file except in compliance\n",
"# with the License. You may obtain a copy of the License at\n",
"#\n",
"# http://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing,\n",
"# software distributed under the License is distributed on an\n",
"# \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY\n",
"# KIND, either express or implied. See the License for the\n",
"# specific language governing permissions and limitations\n",
"# under the License."
],
"metadata": {
"id": "fCjymAKWJiTh"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# this notebook intended for internal testing purpose."
],
"metadata": {
"id": "CCAvj4mQFR5x"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Installing Apache Beam can be installed directly from the main branch of https://github.com/apache/beam or can be installed via `pip install apache_beam>=2.45.0`"
AnandInguva marked this conversation as resolved.
Show resolved Hide resolved
],
"metadata": {
"id": "IL7coD4DJqzG"
}
},
{
"cell_type": "code",
"source": [
"!git clone https://github.com/apache/beam.git\n",
"!pip install -r beam/sdks/python/build-requirements.txt\n",
"!pip install -e beam/sdks/python[gcp]\n",
"!pip install matplotlib\n",
"!pip install pandas"
],
"metadata": {
"id": "yW4okqmpECqY"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"Import necessary dependencies"
],
"metadata": {
"id": "ZPt3DbZcL-Ki"
}
},
{
"cell_type": "code",
"source": [
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from apache_beam.testing.load_tests import load_test_metrics_utils\n",
"from apache_beam.testing.load_tests.load_test_metrics_utils import BigQueryMetricsFetcher"
],
"metadata": {
"id": "xYGgc-tpE9qY"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"bq_project = 'apache-beam-testing'\n",
"bq_dataset = '<bq-dataset-name>' # sample value: beam_run_inference\n",
"bq_table = '<bq-table>' # sample value: torch_inference_imagenet_results_resnet152\n",
"metric_name = '<perf-alerted-metric-name>' # sample value: mean_load_model_latency_milli_secs\n",
"\n",
"query = f\"\"\"\n",
" SELECT *\n",
" FROM {bq_project}.{bq_dataset}.{bq_table}\n",
" WHERE CONTAINS_SUBSTR(({load_test_metrics_utils.METRICS_TYPE_LABEL}), '{metric_name}')\n",
" ORDER BY {load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL} DESC\n",
" LIMIT 30\n",
" \"\"\"\n"
],
"metadata": {
"id": "nyMmUpRrD_zV"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"If the performance/load test store the results in BigQuery using this [schema](https://github.com/apache/beam/blob/83679216cce2d52dbeb7e837f06ca1d57b31d509/sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py#L66),\n",
"then fetch the metric_values for a `metric_name` for the last `30` runs and display a plot using matplotlib.\n"
],
"metadata": {
"id": "RwlsXCLbVs_2"
}
},
{
"cell_type": "code",
"source": [
"big_query_metrics_fetcher = BigQueryMetricsFetcher()\n",
"metric_data: pd.DataFrame = big_query_metrics_fetcher.fetch(query=query)"
],
"metadata": {
"id": "rmOE_odNEBFK"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"# sort the data to view it in chronological order.\n",
"metric_data.sort_values(\n",
" by=[load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL], inplace=True)"
],
"metadata": {
"id": "q-i3qLpGV5Ly"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"metric_data.plot(x=load_test_metrics_utils.SUBMIT_TIMESTAMP_LABEL,\n",
" y=load_test_metrics_utils.VALUE_LABEL)\n",
"plt.show()"
],
"metadata": {
"id": "vbFoxdxHVvtQ"
},
"execution_count": null,
"outputs": []
}
]
}
Loading