Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: run dbt in batches #158

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

feat: run dbt in batches #158

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f16ecf8
feat: process initial sync in batches
njuguna-n Sep 25, 2024
93d67f2
refactor: try alternative batching process
njuguna-n Sep 27, 2024
4314faa
refactor: how we do batch dbt updates
njuguna-n Sep 30, 2024
49c5cc6
feat: run dbt in batch mode only
njuguna-n Oct 4, 2024
e1e3b9c
chore: run e2e test in batches
njuguna-n Oct 4, 2024
a520e30
feat: run in batches or regular run
njuguna-n Oct 4, 2024
8bd222b
chore: use variable
njuguna-n Oct 4, 2024
0f3405f
refactor: handle batch update error and success in different functions
njuguna-n Oct 7, 2024
b5522c9
chore: refactor: use BATCH_STATUS_TABLE variable and remove unused fu…
njuguna-n Oct 7, 2024
2caf09d
feat: check if metadata table exists
njuguna-n Oct 8, 2024
c36ced3
chore: chnge where the table check happens
njuguna-n Oct 8, 2024
4055e3f
feat: rename pythin file, cchange setup function to use one connectio…
njuguna-n Oct 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions dbt/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ FROM python:3 AS testing

RUN pip install flake8

RUN flake8 dbt-run.py
RUN flake8 dbt_run.py

FROM python:3 AS release

Expand All @@ -14,10 +14,10 @@ RUN adduser dbt
USER dbt

WORKDIR /dbt/
COPY dbt-run.py dbt-run.py
COPY dbt_run.py dbt_run.py
COPY .dbt .dbt
COPY dbt_project.yml dbt_project.yml

RUN mkdir old_manifest

CMD ["python3", "/dbt/dbt-run.py"]
CMD ["python3", "/dbt/dbt_run.py"]
142 changes: 0 additions & 142 deletions dbt/dbt-run.py
View file
Open in desktop

This file was deleted.

3 changes: 3 additions & 0 deletions dbt/dbt_project.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,6 @@ target-path: "target"
clean-targets: ["target", "dbt_modules"]
macro-paths: ["macros"]
log-path: "logs"
vars:
start_timestamp: null
batch_size: null
Loading
Loading