Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add entry point #2616

Merged
merged 34 commits into from
Apr 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
95b66a8
feat: add entry_point.py
JoeWang1127 Apr 2, 2024
964d81a
remove cli in generate_repo.py
JoeWang1127 Apr 2, 2024
5b303cb
remove cli in generate_pr_description.py
JoeWang1127 Apr 2, 2024
4f6cedd
add comment
JoeWang1127 Apr 2, 2024
531244d
fix integration test
JoeWang1127 Apr 2, 2024
a6811d5
rename config
JoeWang1127 Apr 2, 2024
14cdfb9
change pr description file path
JoeWang1127 Apr 2, 2024
24f2070
add comments
JoeWang1127 Apr 3, 2024
853b70c
restore CLI
JoeWang1127 Apr 3, 2024
923f149
format code
JoeWang1127 Apr 3, 2024
1cd3859
change input to current_generation_config
JoeWang1127 Apr 3, 2024
ffc16ee
change to optional inputs
JoeWang1127 Apr 4, 2024
87c7d1c
raise exception if current commit is older than baseline commit
JoeWang1127 Apr 4, 2024
9d723b0
add unit test
JoeWang1127 Apr 4, 2024
fd7897a
add error log
JoeWang1127 Apr 4, 2024
ca902ec
convert time to int
JoeWang1127 Apr 4, 2024
4895971
add comment
JoeWang1127 Apr 4, 2024
87c0bdd
extract helper method
JoeWang1127 Apr 4, 2024
69fd09b
refactor tests
JoeWang1127 Apr 4, 2024
d2b3963
add integration test
JoeWang1127 Apr 4, 2024
b4b907b
change if
JoeWang1127 Apr 4, 2024
ce25efa
refactor
JoeWang1127 Apr 4, 2024
8d1d3f6
restore integration test
JoeWang1127 Apr 4, 2024
e155ea0
add unit tests
JoeWang1127 Apr 4, 2024
8de0c3d
add unit tests
JoeWang1127 Apr 4, 2024
e518d3f
test setup
JoeWang1127 Apr 4, 2024
22eb127
restore unit tests
JoeWang1127 Apr 4, 2024
c25e2bd
refactor
JoeWang1127 Apr 4, 2024
cc137b0
change error message
JoeWang1127 Apr 5, 2024
5dbcd2c
fail if current config is null but baseline is not
JoeWang1127 Apr 5, 2024
3d3522b
two commits should be different
JoeWang1127 Apr 5, 2024
280cee5
change doc
JoeWang1127 Apr 5, 2024
881b557
change comment
JoeWang1127 Apr 5, 2024
4de89de
change readme
JoeWang1127 Apr 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 30 additions & 23 deletions library_generation/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Generate a repository containing GAPIC Client Libraries

The script, `generate_repo.py`, allows you to generate a repository containing
GAPIC client libraries (a monorepo, for example, google-cloud-java) from a
configuration file.
The script, `entry_point.py`, allows you to generate a repository containing
GAPIC client libraries with googleapis commit history (a monorepo, for example,
google-cloud-java) from a configuration file.

## Environment

Expand All @@ -17,30 +17,27 @@ In order to generate a version for each library, a versions.txt has to exist
in `repository_path`.
Please refer to [Repository path](#repository-path--repositorypath---optional) for more information.

## Parameters to generate a repository using `generate_repo.py`
## Parameters to generate a repository using `entry_point.py`

### Generation configuration yaml (`generation_config_yaml`)
### Baseline generation configuration yaml (`baseline_generation_config`)

A path to a configuration file containing parameters to generate the repository.
Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository)
for more information.

### Target library API shortname (`target_library_api_shortname`), optional
An absolute or relative path to a generation_config.yaml.
This config file is used for commit history generation, not library
generation.

If specified, the libray whose `api_shortname` equals to `target_library_api_shortname`
will be generated; otherwise all libraries in the configuration file will be
generated.
This can be useful when you just want to generate one library for debugging
purposes.
### Current generation configuration yaml (`current_generation_config`)

The default value is an empty string, which means all libraries will be generated.
An absolute or relative path to a configuration file containing parameters to
generate the repository.
Please refer [Configuration to generate a repository](#configuration-to-generate-a-repository)
for more information.

### Repository path (`repository_path`), optional

The path to where the generated repository goes.

The default value is the current working directory when running the script.
For example, `cd google-cloud-java && python generate_repo.py ...` without
For example, `cd google-cloud-java && python entry_point.py ...` without
specifying the `--repository_path` option will modify the `google-cloud-java`
repository the user `cd`'d into.

Expand All @@ -49,7 +46,9 @@ right version for each library.
Please refer [here](go/java-client-releasing#versionstxt-manifest) for more info
of versions.txt.

## Output of `generate_repo.py`
## Output of `entry_point.py`

### GAPIC libraries

For each module (e.g. `google-cloud-java/java-asset`), the following files/folders
will be created/modified:
Expand All @@ -73,6 +72,11 @@ will be created/modified:
| pom.xml (repo root dir) | Always generated from inputs |
| versions.txt | New entries will be added if they don’t exist |

### googleapis commit history

If both `baseline_generation_config` and `current_generation_config` are
specified, and they contain different googleapis commit, the commit history will
be generated into `pr_description.txt` in the `repository_path`.

## Configuration to generate a repository

Expand Down Expand Up @@ -184,20 +188,23 @@ libraries:
- proto_path: google/cloud/asset/v1p7beta1
```

## An example to generate a repository using `generate_repo.py`
## An example to generate a repository using `entry_point.py`

```bash
# install python module (allows the `library_generation` module to be imported from anywhere)
python -m pip install -r library_generation/requirements.in
# install library_generation module
python -m pip install library_generation
# generate the repository
python -m library_generation/generate_repo.py generate \
--generation-config-yaml=/path/to/config-file \
python -m library_generation/entry_point.py generate \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, we have to remove -m here, this is what tripped me when I run locally, I updated it in my PR

--baseline-generation-config=/path/to/baseline_config_file \
--current-generation-config=/path/to/current_config_file \
--repository-path=/path/to/repository
```

## An example of generated repository using `generate_repo.py`
## An example of generated repository using `entry_point.py`

If you run `generate_repo.py` with the example [configuration](#an-example-of-generation-configuration)
If you run `entry_point.py` with the example [configuration](#an-example-of-generation-configuration)
shown above, the repository structure is:
```
$repository_path
Expand Down
141 changes: 141 additions & 0 deletions library_generation/cli/entry_point.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os

import click as click
from library_generation.generate_pr_description import generate_pr_descriptions
from library_generation.generate_repo import generate_from_yaml
from library_generation.model.generation_config import from_yaml
from library_generation.utils.generation_config_comparator import compare_config


@click.group(invoke_without_command=False)
@click.pass_context
@click.version_option(message="%(version)s")
def main(ctx):
pass


@main.command()
@click.option(
"--baseline-generation-config",
required=False,
default=None,
type=str,
help="""
Absolute or relative path to a generation_config.yaml.
This config file is used for commit history generation, not library
generation.
""",
)
@click.option(
"--current-generation-config",
required=False,
default=None,
type=str,
help="""
Absolute or relative path to a generation_config.yaml that contains the
metadata about library generation.
""",
)
@click.option(
"--repository-path",
type=str,
default=".",
show_default=True,
help="""
The repository path to which the generated files
will be sent.
If not specified, the repository will be generated to the current working
directory.
""",
)
def generate(
baseline_generation_config: str,
current_generation_config: str,
repository_path: str,
):
"""
Compare baseline generation config and current generation config and
generate changed libraries based on current generation config with commit
history.

If baseline generation config is not specified but current generation
config is specified, generate all libraries based on current generation
config without commit history.

If current generation config is not specified but baseline generation
config is specified, raise FileNotFoundError because current generation
config should be the source of truth of library generation.

If both baseline generation config and current generation config are not
specified, generate all libraries based on the default generation config,
which is generation_config.yaml in the current working directory. Raise
FileNotFoundError if the default config does not exist.

The commit history, if generated, will be available in
repository_path/pr_description.txt.
"""
default_generation_config = f"{os.getcwd()}/generation_config.yaml"

if baseline_generation_config is None and current_generation_config is None:
if not os.path.isfile(default_generation_config):
raise FileNotFoundError(
f"{default_generation_config} does not exist. "
"A valid generation config has to be passed in as "
"current_generation_config or exist in the current working "
"directory."
)
current_generation_config = default_generation_config
elif current_generation_config is None:
blakeli0 marked this conversation as resolved.
Show resolved Hide resolved
raise FileNotFoundError(
"current_generation_config is not specified when "
"baseline_generation_config is specified. "
"current_generation_config should be the source of truth of "
"library generation."
)

current_generation_config = os.path.abspath(current_generation_config)
repository_path = os.path.abspath(repository_path)
if not baseline_generation_config:
blakeli0 marked this conversation as resolved.
Show resolved Hide resolved
# Execute full generation based on current_generation_config if
# baseline_generation_config is not specified.
# Do not generate pull request description.
generate_from_yaml(
config=from_yaml(current_generation_config),
repository_path=repository_path,
)
return

# Compare two generation configs and only generate changed libraries.
# Generate pull request description.
baseline_generation_config = os.path.abspath(baseline_generation_config)
config_change = compare_config(
baseline_config=from_yaml(baseline_generation_config),
current_config=from_yaml(current_generation_config),
)
generate_from_yaml(
config=config_change.current_config,
repository_path=repository_path,
target_library_names=config_change.get_changed_libraries(),
)
generate_pr_descriptions(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to rename generate_pr_description to something like generate_commit_history, because we just happen to use PR description to capture the commit history, and we could use other ways. It could be a separate PR.

config=config_change.current_config,
baseline_commit=config_change.baseline_config.googleapis_commitish,
description_path=repository_path,
)


if __name__ == "__main__":
main()
Loading
Loading