Skip to content

Commit

Permalink
[docs-beta] migrate - CI/CD docs (#26508)
Browse files Browse the repository at this point in the history
## Summary & Motivation

Marked the following links as `/todo`:

```
  - Broken link on source page path = /dagster-plus/features/ci-cd/branch-deployments/change-tracking:
     -> linking to /dagster-plus/managing-deployments/branch-deployments/using-branch-deployments
     -> linking to /guides/dagster/asset-versioning-and-caching
     -> linking to /concepts/metadata-tags/tags
     -> linking to /concepts/metadata-tags/asset-metadata#attaching-definition-metadata
  - Broken link on source page path = /dagster-plus/features/ci-cd/branch-deployments/dagster-cloud-cli:
     -> linking to /dagster-plus/account/managing-user-agent-tokens
  - Broken link on source page path = /dagster-plus/features/ci-cd/branch-deployments/testing:
     -> linking to /concepts/assets/software-defined-assets
     -> linking to /concepts/ops-jobs-graphs/ops
     -> linking to /concepts/ops-jobs-graphs/graphs
     -> linking to /concepts/assets/asset-jobs
     -> linking to /concepts/resources
     -> linking to /concepts/io-management/io-managers
     -> linking to /dagster-plus/managing-deployments/branch-deployments/using-branch-deployments-with-github
     -> linking to /dagster-plus/managing-deployments/branch-deployments/using-branch-deployments-with-gitlab
     -> linking to /dagster-plus/account/managing-users/managing-user-roles-permissions
     -> linking to /dagster-plus/managing-deployments/reserved-environment-variables
     -> linking to /dagster-plus/managing-deployments/environment-variables-and-secrets
```

## How I Tested These Changes

## Changelog

> Insert changelog entry or delete this section.
  • Loading branch information
cmpadden authored Dec 20, 2024
1 parent 8be213c commit c03453b
Show file tree
Hide file tree
Showing 13 changed files with 1,020 additions and 8 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,210 @@ sidebar_position: 200
unlisted: true
---

{/* TODO copy from https://docs.dagster.io/dagster-plus/managing-deployments/branch-deployments/change-tracking */}
:::note
This guide is applicable to Dagster+.
:::

Branch Deployments Change Tracking makes it eaiser for you and your team to identify how changes in a pull request will impact data assets. By the end of this guide, you'll understand how Change Tracking works and what types of asset changes can be detected.

## How it works

Branch Deployments compare asset definitions in the branch deployment against the asset definitions in the main deployment. The UI will then mark changed assets accordingly. For example, if the pull request associated with the branch deployment adds a new asset, the UI will display a label indicating the addition.

You can also apply filters to show only new and changed assets in the UI. This makes it easy to understand which assets will be impacted by the changes in the pull request associated with the branch deployment.

{/* **Note:** The default main deployment is `prod`. To configure a different deployment as the main deployment, [create a branch deployment using the dagster-cloud CLI](/dagster-plus/managing-deployments/branch-deployments/using-branch-deployments) and specify it using the optional `--base-deployment-name` parameter. */}
**Note:** The default main deployment is `prod`. To configure a different deployment as the main deployment, [create a branch deployment using the dagster-cloud CLI](/todo) and specify it using the optional `--base-deployment-name` parameter.

## Supported change types

Change Tracking can detect the following changes to assets:

- [New assets](#new-assets)
- [Code versions](#code-versions)
- [Upstream dependencies](#upstream-dependencies)
- [Partitions definitions](#partitions-definitions)
- [Tags](#tags)
- [Metadata](#metadata)

### New assets

If an asset is new in the branch deployment, the asset will have a **New in branch** label in the asset graph:

![Change tracking new](/images/dagster-cloud/managing-deployments/change-tracking-new.png)

### Code versions

If using the `code_version` argument on the asset decorator, Change Tracking can detect when this value changes.

{/* Some Dagster integrations, like `dagster-dbt`, automatically compute code versions for you. For more information on code versions, refer to the [Code versioning guide](/guides/dagster/asset-versioning-and-caching). */}
Some Dagster integrations, like `dagster-dbt`, automatically compute code versions for you. For more information on code versions, refer to the [Code versioning guide](/todo).

<Tabs>
<TabItem value="Asset in the Dagster UI">

In this example, the `customers` asset has a **Changed in branch** label indicating its `code_version` has been changed.

Click the **Asset definition** tab to see the code change that created this label.

![Change tracking code version](/images/dagster-cloud/managing-deployments/change-tracking-code-version.png)

</TabItem>
<TabItem value="Asset definition">

**In the main branch**, we have a `customers` asset with a code version of `v1`:

```python file=/dagster_cloud/branch_deployments/change_tracking_code_version.py startafter=start_main_deployment endbefore=end_main_deployment dedent=4
@asset(code_version="v1")
def customers(): ...
```

**In the pull request**, `customers` is modified to change the code version to `v2`:

```python file=/dagster_cloud/branch_deployments/change_tracking_code_version.py startafter=start_branch_deployment endbefore=end_branch_deployment dedent=4
@asset(code_version="v2")
def customers(): ...
```

</TabItem>
</Tabs>

### Upstream dependencies

Change Tracking can detect when an asset's upstream dependencies have changed, whether they've been added or removed.

**Note**: If an asset is marked as having changed dependencies, it means that the <PyObject object="AssetKey" pluralize /> defining its upstream dependencies have changed. It doesn't mean that an upstream dependency has new data.

<Tabs>
<TabItem value="Asset in the Dagster UI">

In this example, the `returns` asset has a **Changed in branch** label indicating it has changed dependencies.

Click the **Asset definition** tab to see the code change that created this label.

![Change tracking dependencies](/images/dagster-cloud/managing-deployments/change-tracking-dependencies.png)

```python file=/dagster_cloud/branch_deployments/change_tracking_dependencies.py startafter=start_branch_deployment endbefore=end_branch_deployment dedent=4
@asset(deps=[orders, customers])
def returns(): ...
```

</TabItem>
</Tabs>

### Partitions definitions

Change Tracking can detect if an asset's <PyObject object="PartitionsDefinition" /> has been changed, whether it's been added, removed, or updated.

<Tabs>
<TabItem value="Asset in the Dagster UI">

In this example, the `weekly_orders` asset has a **Changed in branch** label indicating a changed partitions definition.

Click the **Asset definition** tab to see the code change that created this label.

![Change tracking partitions](/images/dagster-cloud/managing-deployments/change-tracking-partitions.png)

</TabItem>
<TabItem value="Asset definition">

**In the main branch**, we have a `weekly_orders` asset:

```python file=/dagster_cloud/branch_deployments/change_tracking_partitions_definition.py startafter=start_main_deployment endbefore=end_main_deployment dedent=4
@asset(partitions_def=WeeklyPartitionsDefinition(start_date="2024-01-01"))
def weekly_orders(): ...
```

**In the pull request**, we updated the <PyObject object="WeeklyPartitionsDefinition" /> to start one year earlier:

```python file=/dagster_cloud/branch_deployments/change_tracking_partitions_definition.py startafter=start_branch_deployment endbefore=end_branch_deployment dedent=4
@asset(partitions_def=WeeklyPartitionsDefinition(start_date="2023-01-01"))
def weekly_orders(): ...
```

</TabItem>
</Tabs>

### Tags

{/* Change Tracking can detect when an [asset's tags](/concepts/metadata-tags/tags) have changed, whether they've been added, modified, or removed. */}
Change Tracking can detect when an [asset's tags](/todo) have changed, whether they've been added, modified, or removed.

<Tabs>
<TabItem value="Asset in the Dagster UI">

In this example, the `fruits_in_stock` asset has a **Changed in branch** label indicating it has changed tags.

Click the **Asset definition** tab to see the code change that created this label.

![Change tracking tags](/images/dagster-cloud/managing-deployments/change-tracking-tags.png)

</TabItem>
<TabItem value="Asset definition">

**In the main branch**, we have a `fruits_in_stock` asset:

```python file=/dagster_cloud/branch_deployments/change_tracking_tags.py startafter=start_main_deployment endbefore=end_main_deployment dedent=4
@asset(tags={"section": "produce"})
def fruits_in_stock(): ...
```

**In the pull request**, we added the `type: perishable` tag to `fruits_in_stock`:

```python file=/dagster_cloud/branch_deployments/change_tracking_tags.py startafter=start_branch_deployment endbefore=end_branch_deployment dedent=4
@asset(tags={"section": "produce", "type": "perishable"})
def fruits_in_stock(): ...
```

</TabItem>
</Tabs>

### Metadata

{/* Change Tracking can detect when an [asset's definition metadata](/concepts/metadata-tags/asset-metadata#attaching-definition-metadata) has changed, whether it's been added, modified, or removed. */}
Change Tracking can detect when an [asset's definition metadata](/todo) has changed, whether it's been added, modified, or removed.

<Tabs>
<TabItem value="Asset in the Dagster UI">

In this example, the `produtcs` asset has a **Changed in branch** label indicating it has changed metadata.

Click the **Asset definition** tab to see the code change that created this label.

![Change tracking metadata](/images/dagster-cloud/managing-deployments/change-tracking-metadata.png)

</TabItem>
<TabItem value="Asset definition">

**In the main branch**, we have a `products` asset:

```python file=/dagster_cloud/branch_deployments/change_tracking_metadata.py startafter=start_main_deployment endbefore=end_main_deployment dedent=4
@asset(metadata={"expected_columns": ["sku", "price", "supplier"]})
def products(): ...
```

**In the pull request**, we update the value of the `expected_columns` metadata on `products`:

```python file=/dagster_cloud/branch_deployments/change_tracking_metadata.py startafter=start_branch_deployment endbefore=end_branch_deployment dedent=4
@asset(metadata={"expected_columns": ["sku", "price", "supplier", "backstock"]})
def products(): ...
```

</TabItem>
</Tabs>

## Related

{/*
<ArticleList>
<ArticleListItem
title="Branch deployments"
href="/dagster-plus/managing-deployments/branch-deployments"
></ArticleListItem>
<ArticleListItem
title="Managing Dagster+ deployments"
href="/dagster-plus/managing-deployments"
></ArticleListItem>
<ArticleListItem title="Dagster Cloud" href="/dagster-plus"></ArticleListItem>
</ArticleList>
*/}
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,156 @@ sidebar_position: 300
unlisted: true
---

{/* TODO copy from https://docs.dagster.io/dagster-plus/managing-deployments/dagster-plus-cli#using-the-dagster-cloud-cli */}
:::note
This guide is applicable to Dagster+.
:::

The `dagster-cloud` CLI is a command-line toolkit designed to work with Dagster+.

In this guide, we'll cover how to install and configure the `dagster-cloud` CLI, get help, and use some helpful environment variables and CLI options.

## Installing the CLI

The Dagster+ Agent library is available in PyPi. To install, run:

```shell
pip install dagster-cloud
```

Refer to the [configuration section](#configuring-the-cli) for next steps.

### Completions

Optionally, you can install command-line completions to make using the `dagster-cloud` CLI easier.

To have the CLI install these completions to your shell, run:

```shell
dagster-cloud --install-completion
```

To print out the completion for copying or manual installation:

```shell
dagster-cloud --show-completion
```

## Configuring the CLI

The recommended way to set up your CLI's config for long-term use is through the configuration file, located by default at `~/.dagster_cloud_cli/config`.

### Setting up the configuration file

Set up the config file:

```shell
dagster-cloud config setup
```

Select your authentication method. **Note**: Browser authentication is the easiest method to configure.

<details>
<summary><strong>BROWSER AUTHENTICATION</strong></summary>

The easiest way to set up is to authenticate through the browser.

```shell
$ dagster-cloud config setup
? How would you like to authenticate the CLI? (Use arrow keys)
» Authenticate in browser
Authenticate using token
Authorized for organization `hooli`

? Default deployment: prod
```

When prompted, you can specify a default deployment. If specified, a deployment won't be required in subsequent `dagster-cloud` commands. The default deployment for a new Dagster+ organization is `prod`.

</details>

<details>
<summary><strong>TOKEN AUTHENTICATION</strong></summary>

{/* Alternatively, you may authenticate using a user token. Refer to the [Managing user and agent tokens guide](/dagster-plus/account/managing-user-agent-tokens) for more info. */}
Alternatively, you may authenticate using a user token. Refer to the [Managing user and agent tokens guide](/todo) for more info.

```shell
$ dagster-cloud config setup
? How would you like to authenticate the CLI? (Use arrow keys)
Authenticate in browser
» Authenticate using token

? Dagster+ organization: hooli
? Dagster+ user token: *************************************
? Default deployment: prod
```

When prompted, specify the following:

- **Organization** - Your organization name as it appears in your Dagster+ URL. For example, if your Dagster+ instance is `https://hooli.dagster.cloud/`, this would be `hooli`.
- **User token** - The user token.
- **Default deployment** - **Optional**. A default deployment. If specified, a deployment won't be required in subsequent `dagster-cloud` commands. The default deployment for a new Dagster+ organization is `prod`.

</details>

### Viewing and modifying the configuration file

To view the contents of the CLI configuration file, run:

```shell
$ dagster-cloud config view

default_deployment: prod
organization: hooli
user_token: '*******************************8214fe'
```

Specify the `--show-token` flag to show the full user token.

To modify the existing config, re-run:

```shell
dagster-cloud config setup
```

## Toggling between deployments

To quickly toggle between deployments, run:

```shell
dagster-cloud config set-deployment <deployment_name>
```

## Getting help

To view help options in the CLI:

```shell
dagster-cloud --help
```

## Reference

- [Custom configuration file path](#custom-configuration-file-path)
- [Environment variables and CLI options](#environment-variables-and-cli-options)

### Custom configuration file path

Point the CLI at an alternate config location by specifying the `DAGSTER_CLOUD_CLI_CONFIG` environment variable.

### Environment variables and CLI options

Environment variables and CLI options can be used in place of or to override the CLI configuration file.

The priority of these items is as follows:

- **CLI options** - highest
- **Environment variables**
- **CLI configuration** - lowest

| Setting | Environment variable | CLI flag | CLI config value |
| ------------ | ---------------------------- | ---------------------- | -------------------- |
| Organization | `DAGSTER_CLOUD_ORGANIZATION` | `--organization`, `-o` | `organization` |
| Deployment | `DAGSTER_CLOUD_DEPLOYMENT` | `--deployment`, `-d` | `default_deployment` |
| User Token | `DAGSTER_CLOUD_API_TOKEN` | `--user-token`, `-u` | `user_token` |

Loading

0 comments on commit c03453b

Please sign in to comment.