Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(#1720): add instructions for running dbt locally #1721

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat(#1720): add instructions for running dbt locally #1721

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion content/en/hosting/analytics/building-dbt-models.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,10 @@ packages:
```
To avoid breaking changes in downstream models, include `revision` in the dependency, which should be a version tag for `cht-pipeline`.

In the CHT Sync config, set the URL of dbt GitHub repository to the `CHT_PIPELINE_BRANCH_URL` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}), either in `.env` if using Docker compose, or in `values.yaml` if using Kubernetes.
It is helpful to test changes locally before commiting them to a remote repository.
For local testing and devleopment, in the CHT Sync config, set the path to the dbt project to the `DBT_LOCAL_PATH` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}) in `.env`
Use the [docker compose setup]({{< relref "hosting/analytics/setup-docker-compose" >}}) for CHT Sync and use `docker-compose.local.yml` instead of `docker-compose.yml`.
For production and remote deployments, in the CHT Sync config, set the URL of dbt GitHub repository to the `CHT_PIPELINE_BRANCH_URL` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}), either in `.env` if using Docker compose, or in `values.yaml` if using Kubernetes.

{{% alert title="Note" %}}
If `CHT_PIPELINE_BRANCH_URL` is pointing to a private GitHub repository, you'll need an access token in the URL. Assuming your repository is `medic/cht-pipeline`, you would replace `<PAT>` with an access token: `https://<PAT>@github.com/medic/cht-pipeline.git#main`. Please see [GitHub's instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) on how to generate a token. If you create a fine-grained access token you need to provide read and write access to the [contents](https://docs.github.com/en/rest/authentication/permissions-required-for-fine-grained-personal-access-tokens?apiVersion=2022-11-28#repository-permissions-for-contents) of the repository.
Expand Down
17 changes: 9 additions & 8 deletions content/en/hosting/analytics/environment-variables.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ All the variables in the `.env` file:
| `POSTGRES_SCHEMA` | `v1` | PostgreSQL schema |
| `POSTGRES_TABLE` | `couchdb` | PostgreSQL table where the CouchDB data is copied |
| `POSTGRES_HOST` | `postgres` | PostgreSQL instance |
| `POSTGRES_PORT` | `5432` | PostgreSQL port |
| `CHT_PIPELINE_BRANCH_URL` | `"https://github.com/medic/cht-pipeline.git#main"` | cht-pipeline branch containing the dbt models |
| `POSTGRES_PORT` | `5432` | PostgreSQL port |
| `CHT_PIPELINE_BRANCH_URL` | `"https://github.com/medic/cht-pipeline.git#main"` | cht-pipeline branch containing the dbt models |
| `DBT_LOCAL_PATH` | `""` | if running dbt locally, path to the dbt project on the host |
| `DATAEMON_INTERVAL` | `5` | Interval (in minutes) for looking for new changes in the CouchDB data |
| `COUCHDB_USER` | `medic` | Username of the CouchDB instance |
| `COUCHDB_PASSWORD` | `password` | Password of the CouchDB instance |
| `COUCHDB_DBS` | `"medic"` | Comma separated list of databases to sync e.g `"medic, medic_sentinel"` |
| `COUCHDB_HOST` | `couchdb` | Host of the CouchDB instance |
| `COUCHDB_PORT` | `5984` | Port of the CouchDB instance |
| `COUCHDB_SECURE` | `false` | Is connection to CouchDB instance secure? |
| `COUCHDB_USER` | `medic` | Username of the CouchDB instance |
| `COUCHDB_PASSWORD` | `password` | Password of the CouchDB instance |
| `COUCHDB_DBS` | `"medic"` | Comma separated list of databases to sync e.g `"medic, medic_sentinel"` |
| `COUCHDB_HOST` | `couchdb` | Host of the CouchDB instance |
| `COUCHDB_PORT` | `5984` | Port of the CouchDB instance |
| `COUCHDB_SECURE` | `false` | Is connection to CouchDB instance use HTTPS? |

9 changes: 9 additions & 0 deletions content/en/hosting/analytics/setup-docker-compose.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,15 @@ For production CHT Core deployments, the port will most likely need to be set to
The first time you run the commands from any of the sections below it will need to download many Docker images and will take a while. You'll know it's done when you see `#8 DONE 0.0s` and you are returned to the command line. Be patient!
{{% /alert %}}

### Running DBT models locally

When developing DBT models, it is helpful to test changes locally before commiting them to a remote repository.
Set the path to the project to the `DBT_LOCAL_PATH` [environment variable]({{< relref "hosting/analytics/environment-variables" >}}) in `.env` and use `docker-compose.local.yml`.

```sh
docker compose -f docker-compose.postgres.yml -f docker-compose.local.yml up -d
```

### Separate CouchDB instance

This setup involves starting couch2pg, PostgreSQL, pgAdmin and dbt. It assumes you have a CouchDB instance running, and you updated the `.env` CouchDB variables accordingly.
Expand Down
Loading