Skip to content

Commit

Permalink
[AUTO-MERGE] Merge main into develop via merge-main-to-develop (#2654)
Browse files Browse the repository at this point in the history
* Remove unnecessary files and subheaders in documentation (#2563)

Tidying up.

* Leverage PEP-585 (#2540)

Signed-off-by: Merel Theisen <[email protected]>
Co-authored-by: Jannic <[email protected]>

* [READY] Bring deployment docs up-to-date and add new pages for additional targets (#2557)

Ready at last. Still some conversation to be had about how we reflect "confidence" in the plugins we describe (what version they were last tested against) but the text is ready and better than what's there currently, so let's go!

* Remove mentions to `.egg` and stop producing them from `kedro package` (#2568)

* Stop producing `.egg` files from `kedro package`

Fix gh-2273.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Upgrade import-linter

Fix gh-2570.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Partially remove reliance on setup.py for micropackaging

Incomplete effort, see gh-2414.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Revert "Partially remove reliance on setup.py for micropackaging"

This reverts commit fd3efa8.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

---------

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* OmegaConfigLoader returns `Dict` instead of `DictConfig`, resolves runtime_params properly (#2467)

* Fix typehint

Signed-off-by: Nok Chan <[email protected]>

* test push

Signed-off-by: Nok Chan <[email protected]>

* POC of fix to solve the runtime param resolution problem

Signed-off-by: Nok Chan <[email protected]>

* Fix OmegaConfigLoadaer - resolve runtime_params early

Signed-off-by: Nok Chan <[email protected]>

* Delegate the intialization of runtime_params to AbstractConfigLoader

Signed-off-by: Nok Chan <[email protected]>

* Add test for interpolated value and globals

Signed-off-by: Nok Chan <[email protected]>

* add more test and linting

Signed-off-by: Nok Chan <[email protected]>

* refactor

Signed-off-by: Nok Chan <[email protected]>

* update release note

Signed-off-by: Nok Chan <[email protected]>

* Apply comments and refactor the test

Signed-off-by: Nok Chan <[email protected]>

* Update RELEASE.md

Co-authored-by: Antony Milne <[email protected]>

---------

Signed-off-by: Nok Chan <[email protected]>
Co-authored-by: Antony Milne <[email protected]>

* Revert special logic handling DictConfig (#2582)

* Partially revert #2378 as OCL no longer returning DictConfig

Signed-off-by: Nok Chan <[email protected]>

* Revert #2176 with special logging handling

Signed-off-by: Nok Chan <[email protected]>

* remove unused import

Signed-off-by: Nok Chan <[email protected]>

---------

Signed-off-by: Nok Chan <[email protected]>

* Re-title and modify introduction to databricks testing guide in contribution section (#2579)

* modify databricks testing guide for contributors

Signed-off-by: Jo Stichbury <[email protected]>

* Update docs/source/contribution/development_for_databricks.md

Co-authored-by: Jannic <[email protected]>

* Update development_for_databricks.md

---------

Signed-off-by: Jo Stichbury <[email protected]>
Co-authored-by: Jannic <[email protected]>

* Quick fix .gitpod.yml (#2585)

* `OmegaConfigLoader` should not show `MissingConfigException` when the file is empty (#2584)

* Allow empty config files

Signed-off-by: Ankita Katiyar <[email protected]>

* Remove condition for empty config

Signed-off-by: Ankita Katiyar <[email protected]>

* Revert "Remove condition for empty config"

This reverts commit 7961155.

* Update release notes

Signed-off-by: Ankita Katiyar <[email protected]>

* Add unit test

Signed-off-by: Ankita Katiyar <[email protected]>

---------

Signed-off-by: Ankita Katiyar <[email protected]>

* Proposal of updated settings.py (#2587)

* Proposal of updated settings.py

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Update settings after comment

Signed-off-by: Nok <[email protected]>

---------

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Nok <[email protected]>

* Collaborative Experiment Tracking Docs  (#2589)

* Add metadata attribute to `kedro.io` datasets (#2537)

* Add metadata attribute to kedro.io datasets

Signed-off-by: Ahdra Merali <[email protected]>

* Update specs in pipeline hooks docstrings (#2598)

* Update pipeline hooks docstrings

Signed-off-by: Tomas Van Pottelbergh <[email protected]>

* Acknowledge community contribution

---------

Signed-off-by: Tomas Van Pottelbergh <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Miscellaneous hooks improvements (#2596)

* Add debug level logging for hook execution order

Fix gh-1942.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Clarify role of setuptools in hooks

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Remove mentions to outdated and unused `pkg_resources`

Fix gh-2391.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Adapt example hook to spec signature

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Grammar and style fixes in docs

Co-authored-by: Jo Stichbury <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Revert "Adapt example hook to spec signature"

This reverts commit ac1b495.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

---------

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>

* Minor modification to titles of deployment docs to replace "deprecated" with "legacy" (#2606)

* FIX: Typo in markdown for kedro notebooks docs (#2610)

* FIX: Typo in markdown for kedro notebooks docs

Signed-off-by: debugger24 <[email protected]>

* Acknowledge community contribution

---------

Signed-off-by: debugger24 <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Add the Kedro RichHandler (#2592)

* Add the Kedro RichHandler

Signed-off-by: Nok Chan <[email protected]>

* Clean up imports and comments

Signed-off-by: Nok Chan <[email protected]>

* Map arguments to add flexibility to customize arguments available in RichHandler

Signed-off-by: Nok Chan <[email protected]>

* Clean up rich_logger.py and unused imports

Signed-off-by: Nok Chan <[email protected]>

* Add docs - first iteration

Signed-off-by: Nok Chan <[email protected]>

* Fix imports and small refactoring

Signed-off-by: Nok Chan <[email protected]>

* Improve the doc

Signed-off-by: Nok Chan <[email protected]>

* Update the project's template

Signed-off-by: Nok Chan <[email protected]>

* format docstring

Signed-off-by: Nok Chan <[email protected]>

* Apply Jo's comment on the doc

Signed-off-by: Nok Chan <[email protected]>

* Fix the link with RichHandler

Signed-off-by: Nok Chan <[email protected]>

* Apply suggestions from code review

Co-authored-by: Antony Milne <[email protected]>

* Fix broken doc

Signed-off-by: Nok Chan <[email protected]>

* update template defaults

Signed-off-by: Nok Chan <[email protected]>

* refactoring RichHandler and improve doc

Signed-off-by: Nok Chan <[email protected]>

* Fix broken test due to new default RichHandler

Signed-off-by: Nok Chan <[email protected]>

* Fix doc

Signed-off-by: Nok Chan <[email protected]>

* Move kedro.extras.logging.RichHandler -> kedro.logging.RichHandler

Signed-off-by: Nok Chan <[email protected]>

* add kedro/logging.py and remove the unnecessary import that cause namespace collision

Signed-off-by: Nok Chan <[email protected]>

* Refactor test

Signed-off-by: Nok Chan <[email protected]>

* Add more tests

Signed-off-by: Nok Chan <[email protected]>

* Fix lots of import, docstring

Signed-off-by: Nok Chan <[email protected]>

* Reorder docstring

Signed-off-by: Nok Chan <[email protected]>

* Fix tests

Signed-off-by: Nok Chan <[email protected]>

* Link options to doc

Signed-off-by: Nok Chan <[email protected]>

* Clean up doc and unused logger

Signed-off-by: Nok Chan <[email protected]>

* Fix spelling `customise`

Signed-off-by: Nok Chan <[email protected]>

* Refactor test with fixture

Signed-off-by: Nok Chan <[email protected]>

* Remove unnecessary copy() because it is now a fixture

Signed-off-by: Nok Chan <[email protected]>

* Split up test_rich_traceback_configuration() to smaller tests

Signed-off-by: Nok Chan <[email protected]>

* Add RELEASE.md

Signed-off-by: Nok Chan <[email protected]>

* Update doc

Signed-off-by: Nok Chan <[email protected]>

* Fix template logging

* Mention awesome-kedro in docs (#2620)

* Mention awesome-kedro in docs

See kedro-org/kedro-devrel#40

* Update CONTRIBUTING.md

* Spelling and style fixes

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

Co-authored-by: Jo Stichbury <[email protected]>

---------

Co-authored-by: Jo Stichbury <[email protected]>

* Update SnowparkTableDataSet docs (#2485)

* Update SnowparkTableDataSet docs

* Add missing snowpark dependencies for docs building

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

---------

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Release 0.18.9 (#2619)

* Update __init__.py

* Update release note

Signed-off-by: Nok <[email protected]>

* Update Version Numbers across docs

Signed-off-by: Nok <[email protected]>

* Update RELEASE.md

* Update Release Note

Signed-off-by: Nok <[email protected]>

---------

Signed-off-by: Nok <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>

* Add documentation for deploying packaged Kedro projects on Databricks (#2595)

* Add deployment workflow page

Signed-off-by: Jannic Holzer <[email protected]>

* Add table of contents, entry point guide, data and conf upload guide

Signed-off-by: Jannic Holzer <[email protected]>

* Add detailed instructions for creating a job on Databricks

Signed-off-by: Jannic Holzer <[email protected]>

* Add images and automated deployment resources

Signed-off-by: Jannic Holzer <[email protected]>

* Remove use of 'allows', add summary

Signed-off-by: Jannic Holzer <[email protected]>

* Remove link to missing image

Signed-off-by: Jannic Holzer <[email protected]>

* Add deployment workflow to toctree

Signed-off-by: Jannic Holzer <[email protected]>

* Lint and fix missing link

Signed-off-by: Jannic Holzer <[email protected]>

* Minor style, syntax and grammar improvements

Signed-off-by: Jannic Holzer <[email protected]>

* Fixes for correctness during validation

Signed-off-by: Jannic Holzer <[email protected]>

* Add instructions for creating log output location

Signed-off-by: Jannic Holzer <[email protected]>

* Lint

Signed-off-by: Jannic Holzer <[email protected]>

* Lint databricks_run

Signed-off-by: Jannic Holzer <[email protected]>

* Minor wording change in reference to logs

Signed-off-by: Jannic Holzer <[email protected]>

* Modify reference to Pyspark-Iris

Signed-off-by: Jannic Holzer <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Fix linter errors to enable docs build for inspection

Signed-off-by: Jo Stichbury <[email protected]>

* Update build-docs.sh

* Fix broken link

Signed-off-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Remove spurious word

Signed-off-by: Jannic Holzer <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Add advantages subheading

Signed-off-by: Jannic Holzer <[email protected]>

* Update docs/source/integrations/databricks_deployment_workflow.md

Co-authored-by: Jo Stichbury <[email protected]>

* Add alternative ways to upload data to DBFS

Signed-off-by: Jannic Holzer <[email protected]>

* Move note on unpackaged config and data

Signed-off-by: Jannic Holzer <[email protected]>

* Fix broken links

Signed-off-by: Jannic Holzer <[email protected]>

* Move databricks back into deployment section

Signed-off-by: Jo Stichbury <[email protected]>

* Remove references to PySpark Iris (pyspark-iris) starter

Signed-off-by: Jannic Holzer <[email protected]>

* Graphics links fixes, revise titles

Signed-off-by: Jo Stichbury <[email protected]>

* Fix broken internal link

Signed-off-by: Jo Stichbury <[email protected]>

* Fix links broken by new folder

Signed-off-by: Jo Stichbury <[email protected]>

* Remove logs directory

Signed-off-by: Jannic Holzer <[email protected]>

* Update image of final job configuration

Signed-off-by: Jannic Holzer <[email protected]>

* Add full stops in list.

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix conda environment name.

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Modify wording and image for creating a new job cluster

Signed-off-by: Jannic Holzer <[email protected]>

* Modify wording in guide to create new job cluster

Signed-off-by: Jannic Holzer <[email protected]>

* Remove --upgrade option

Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Add both ways of creating a new job

Signed-off-by: Jannic Holzer <[email protected]>

---------

Signed-off-by: Jannic Holzer <[email protected]>
Signed-off-by: Jo Stichbury <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Update docs font family to Inter and swap the logo (#2611)

* Update docs font family to Inter

Signed-off-by: Tynan DeBold <[email protected]>

* Match weights to previous Google font

Signed-off-by: Tynan DeBold <[email protected]>

* Update logo images; revert css formatting for easier PR review

Signed-off-by: Tynan DeBold <[email protected]>

* Update text styles

Signed-off-by: Tynan DeBold <[email protected]>

* Add trailing whitespace (?)

Signed-off-by: Tynan DeBold <[email protected]>

* Update Kedro image path

Signed-off-by: Tynan DeBold <[email protected]>

* Update font size

Signed-off-by: Tynan DeBold <[email protected]>

* update font to font-family: 'Inter', sans-serif;

Signed-off-by: huongg <[email protected]>

* testing with important for font-family

Signed-off-by: huongg <[email protected]>

* Make project name lowercase in docs

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* revert back to kedro/develop

Signed-off-by: huongg <[email protected]>

---------

Signed-off-by: Tynan DeBold <[email protected]>
Signed-off-by: huongg <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Huong Nguyen <[email protected]>
Co-authored-by: huongg <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>

* Enable variable interpolation for the catalog with `OmegaConfigLoader` (#2621)

* Enable variable interpolation in catalog by escaping _

Signed-off-by: Merel Theisen <[email protected]>

* Add archive link on docs home page (#2636)

Signed-off-by: Jo Stichbury <[email protected]>

* add ravi-kumar-pilla to tsc (#2629)

Signed-off-by: ravi-kumar-pilla <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>

* Add file for search console verification (#2631)

* add verification file and change to conf

Signed-off-by: Jo Stichbury <[email protected]>

* Fix conf.py so google file is moved to root of docs build

Signed-off-by: Jo Stichbury <[email protected]>

---------

Signed-off-by: Jo Stichbury <[email protected]>

* Chore/update assets for docs (#2625)

* Update docs font family to Inter

Signed-off-by: Tynan DeBold <[email protected]>

* Match weights to previous Google font

Signed-off-by: Tynan DeBold <[email protected]>

* Update logo images; revert css formatting for easier PR review

Signed-off-by: Tynan DeBold <[email protected]>

* Update text styles

Signed-off-by: Tynan DeBold <[email protected]>

* Add trailing whitespace (?)

Signed-off-by: Tynan DeBold <[email protected]>

* Update Kedro image path

Signed-off-by: Tynan DeBold <[email protected]>

* Update font size

Signed-off-by: Tynan DeBold <[email protected]>

* update font to font-family: 'Inter', sans-serif;

Signed-off-by: huongg <[email protected]>

* testing with important for font-family

Signed-off-by: huongg <[email protected]>

* update assets for experiment tracking, docs, pipeline, focus mode, matplotlib and plotly

* update modular pipelines section

Signed-off-by: huongg <[email protected]>

* autoreload

Signed-off-by: huongg <[email protected]>

* reupload autoreload gif

Signed-off-by: huongg <[email protected]>

* revert back to develop branch

Signed-off-by: huongg <[email protected]>

* gif for ET comparison

Signed-off-by: huongg <[email protected]>

* remove jupyter notebook screenshot since its not used anywhere

Signed-off-by: huongg <[email protected]>

* Update databricks viz demo

Signed-off-by: Jannic Holzer <[email protected]>

* pipeline_show_metrics gif

Signed-off-by: huongg <[email protected]>

* update different gif format

Signed-off-by: huongg <[email protected]>

* point to main instead of develop for kedro-banner

Signed-off-by: huongg <[email protected]>

---------

Signed-off-by: Tynan DeBold <[email protected]>
Signed-off-by: huongg <[email protected]>
Signed-off-by: Jannic Holzer <[email protected]>
Co-authored-by: Tynan DeBold <[email protected]>
Co-authored-by: ravi-kumar-pilla <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
Co-authored-by: Jannic Holzer <[email protected]>

* Fix the level heading for collab exp tracking so it shows in sidebar (#2647)

Signed-off-by: Jo Stichbury <[email protected]>

* Update Kedro icons in docs with for rebrand (#2652)

Signed-off-by: Tynan DeBold <[email protected]>

* Modify logos (#2651)

Signed-off-by: Jannic Holzer <[email protected]>

* Fix lint

Signed-off-by: Merel Theisen <[email protected]>

* remove the subsection of kedro.extras.extension

Signed-off-by: Nok <[email protected]>

* Fix lint

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Nok <[email protected]>

* remove the subsection of kedro.extras.extension

Signed-off-by: Nok <[email protected]>

---------

Signed-off-by: Merel Theisen <[email protected]>
Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Nok Chan <[email protected]>
Signed-off-by: Jo Stichbury <[email protected]>
Signed-off-by: Ankita Katiyar <[email protected]>
Signed-off-by: Nok <[email protected]>
Signed-off-by: Ahdra Merali <[email protected]>
Signed-off-by: Tomas Van Pottelbergh <[email protected]>
Signed-off-by: debugger24 <[email protected]>
Signed-off-by: Jannic Holzer <[email protected]>
Signed-off-by: Tynan DeBold <[email protected]>
Signed-off-by: huongg <[email protected]>
Signed-off-by: ravi-kumar-pilla <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
Co-authored-by: Jannic <[email protected]>
Co-authored-by: Juan Luis Cano Rodríguez <[email protected]>
Co-authored-by: Nok Lam Chan <[email protected]>
Co-authored-by: Antony Milne <[email protected]>
Co-authored-by: Ankita Katiyar <[email protected]>
Co-authored-by: rashidakanchwala <[email protected]>
Co-authored-by: Ahdra Merali <[email protected]>
Co-authored-by: Tomas Van Pottelbergh <[email protected]>
Co-authored-by: Rahul Kumar <[email protected]>
Co-authored-by: Tynan DeBold <[email protected]>
Co-authored-by: Huong Nguyen <[email protected]>
Co-authored-by: huongg <[email protected]>
Co-authored-by: Ravi Kumar Pilla <[email protected]>
Co-authored-by: Jannic Holzer <[email protected]>
Co-authored-by: Merel Theisen <[email protected]>
  • Loading branch information
18 people authored Jun 7, 2023
1 parent 2f24de6 commit ab1f7c8
Show file tree
Hide file tree
Showing 164 changed files with 1,625 additions and 1,283 deletions.
1 change: 1 addition & 0 deletions .gitpod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ tasks:
export PIP_USER=no
cd /workspace
pip install -e /workspace/kedro --no-deps
source ~/.bashrc
yes project | kedro new -s pandas-iris --checkout main
command: |
pip install -e /workspace/kedro --no-deps
Expand Down
4 changes: 2 additions & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,6 @@ authors:
- family-names: Theisen
given-names: Merel
title: Kedro
version: 0.18.8
date-released: 2023-05-02
version: 0.18.9
date-released: 2023-05-31
url: https://github.com/kedro-org/kedro
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The Kedro team pledges to foster and maintain a friendly community. We enforce a

You can find the Kedro community on our [Slack organisation](https://slack.kedro.org/), which is where we share news and announcements, and general chat. You're also welcome to post links here to any articles or videos about Kedro that you create, or find, such as how-tos, showcases, demos, blog posts or tutorials.

We also curate a [Github repo that lists content created by the Kedro community](https://github.com/kedro-org/kedro-community).
We also curate a [GitHub repo that lists content created by the Kedro community](https://github.com/kedro-org/awesome-kedro).

## Contribute to the project

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
![Kedro Logo Banner](https://raw.githubusercontent.com/kedro-org/kedro/develop/static/img/kedro_banner.png)
![Kedro Logo Banner](https://raw.githubusercontent.com/kedro-org/kedro/main/static/img/kedro_banner.png)
[![Python version](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9%20%7C%203.10-blue.svg)](https://pypi.org/project/kedro/)
[![PyPI version](https://badge.fury.io/py/kedro.svg)](https://pypi.org/project/kedro/)
[![Conda version](https://img.shields.io/conda/vn/conda-forge/kedro.svg)](https://anaconda.org/conda-forge/kedro)
Expand Down
44 changes: 37 additions & 7 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,43 @@
### DataSets
* If you use `APIDataSet`, move all `requests` specific arguments (e.g. `params`, `headers`), except for `url` and `method`, to under `load_args`.

# Upcoming Release 0.18.9
# Upcoming Release 0.18.10

## Major features and improvements
* Added support for variable interpolation in the catalog with the `OmegaConfigLoader`.

## Bug fixes and other changes

## Breaking changes to the API

## Upcoming deprecations for Kedro 0.19.0

# Release 0.18.9

## Major features and improvements
* `kedro run --params` now updates interpolated parameters correctly when using `OmegaConfigLoader`.
* Added `metadata` attribute to `kedro.io` datasets. This is ignored by Kedro, but may be consumed by users or external plugins.
* Added `kedro.logging.RichHandler`. This replaces the default `rich.logging.RichHandler` and is more flexible, user can turn off the `rich` traceback if needed.

## Bug fixes and other changes
* `OmegaConfigLoader` will return a `dict` instead of `DictConfig`.
* `OmegaConfigLoader` does not show a `MissingConfigError` when the config files exist but are empty.

## Documentation changes
* Added documentation for collaborative experiment tracking within Kedro-Viz.
* Revised section on deployment to better organise content and reflect how recently docs have been updated.
* Minor improvements to fix typos and revise docs to align with engineering changes.

## Breaking changes to the API
* `kedro package` does not produce `.egg` files anymore, and now relies exclusively on `.whl` files.

## Community contributions
Many thanks to the following Kedroids for contributing PRs to this release:

* [tomasvanpottelbergh](https://github.com/tomasvanpottelbergh)
* [https://github.com/debugger24](https://github.com/debugger24)

## Upcoming deprecations for Kedro 0.19.0

# Release 0.18.8

Expand Down Expand Up @@ -143,12 +173,12 @@ Many thanks to the following Kedroids for contributing PRs to this release:
* You can configure config file patterns through `settings.py` without creating a custom config loader.
* Added the following new datasets:

| Type | Description | Location |
| ------------------------------------ | -------------------------------------------------------------------------- | ----------------------------- |
| `svmlight.SVMLightDataSet` | Work with svmlight/libsvm files using scikit-learn library | `kedro.extras.datasets.svmlight` |
| `video.VideoDataSet` | Read and write video files from a filesystem | `kedro.extras.datasets.video` |
| `video.video_dataset.SequenceVideo` | Create a video object from an iterable sequence to use with `VideoDataSet` | `kedro.extras.datasets.video` |
| `video.video_dataset.GeneratorVideo` | Create a video object from a generator to use with `VideoDataSet` | `kedro.extras.datasets.video` |
| Type | Description | Location |
| ------------------------------------ | -------------------------------------------------------------------------- | -------------------------------- |
| `svmlight.SVMLightDataSet` | Work with svmlight/libsvm files using scikit-learn library | `kedro.extras.datasets.svmlight` |
| `video.VideoDataSet` | Read and write video files from a filesystem | `kedro.extras.datasets.video` |
| `video.video_dataset.SequenceVideo` | Create a video object from an iterable sequence to use with `VideoDataSet` | `kedro.extras.datasets.video` |
| `video.video_dataset.GeneratorVideo` | Create a video object from a generator to use with `VideoDataSet` | `kedro.extras.datasets.video` |
* Implemented support for a functional definition of schema in `dask.ParquetDataSet` to work with the `dask.to_parquet` API.

## Bug fixes and other changes
Expand Down
1 change: 1 addition & 0 deletions dependency/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
anyconfig~=0.10.0
attrs>=21.3
build
cachetools~=5.3
click<9.0
cookiecutter>=2.1.1, <3.0
Expand Down
32 changes: 21 additions & 11 deletions docs/source/_static/css/qb1-sphinx-rtd.css
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
@import url("https://fonts.googleapis.com/css?family=Titillium+Web:300,400,600");
@import url("https://fonts.googleapis.com/css2?family=Inter:wght@300;400;600&display=swap");

html, body.wy-body-for-nav {
margin: 0;
padding: 0;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
font-family: 'Titillium Web', sans-serif;
font-family: 'Inter', sans-serif !important;
font-weight: 400;
line-height: 2rem;
}
Expand All @@ -19,7 +19,7 @@ html {
}

body.wy-body-for-nav {
font-size: 1.6rem;
font-size: 1.4289rem;
background: rgb(250, 250, 250) !important;
color: #1e1e21;
position: relative;
Expand All @@ -36,8 +36,8 @@ body.wy-body-for-nav {
padding: 6px 12px;
color: #666;
background-color: #fff;
font-family: inherit;
font-size: 1.6rem;
font-family: 'Inter', sans-serif !important;
font-size: 1.4289rem;
border: 1px #ccc solid;
border-radius: 2px;
transition: all ease 0.15s;
Expand Down Expand Up @@ -116,7 +116,7 @@ body.wy-body-for-nav {
.wy-body-for-nav .wy-menu-vertical p.caption {
margin: 1.5em 0 0.2em;
padding: 0;
font-size: 2rem;
font-size: 1.4289rem;
color: #161616;
font-weight: normal;
text-transform: none;
Expand All @@ -143,15 +143,15 @@ body.wy-body-for-nav {
.wy-body-for-nav .wy-menu-vertical li a {
display: block;
margin: 0;
padding: 1.1rem 0 !important;
font-size: 1.6rem;
padding: 0.9rem 0 !important;
font-size: 1.4289rem;
line-height: 1.2;
color: #222;
background: none !important;
}

.wy-menu-vertical li.toctree-l1>a {
font-size: 1.8rem;
font-size: 1.4289rem;
}

.rst-content.style-external-links a.reference.external:after {
Expand Down Expand Up @@ -309,7 +309,7 @@ article ul li {
}

h1, h2, .rst-content .toctree-wrapper p.caption, h3, h4, h5, h6, legend {
font-family: 'Titillium Web', sans-serif;
font-family: 'Inter', sans-serif !important;
margin: 1.2em 0 1em 0;
line-height: 1.2em;
color: #272c2e;
Expand Down Expand Up @@ -373,7 +373,7 @@ footer span.commit .rst-content tt,
}

.wy-body-for-nav .rst-content a, .wy-body-for-nav footer a {
font-family: inherit;
font-family: 'Inter', sans-serif !important;
font-size: inherit;
color: #006ea7;
text-decoration: none;
Expand Down Expand Up @@ -424,6 +424,16 @@ footer span.commit .rst-content tt,
color: black !important;
}

.wy-side-scroll p.caption[role="heading"] span.caption-text {
font-weight: 700;
}

.wy-nav-content h1,
.wy-nav-content p.caption[role="heading"] .caption-text {
font-size: 1.7857rem;
font-weight: 700;
}


@media screen and (max-width: 768px) {
.wy-body-for-nav .wy-nav-side {
Expand Down
11 changes: 11 additions & 0 deletions docs/source/_static/css/theme-overrides.css
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,14 @@ html.writer-html5 .rst-content .wy-table-responsive table.docutils td > p {
font-size: 1em;
line-height: 1em;
}

img[alt^="mermaid-"] {
max-width: 600px;
}

.rst-content .important {
background: #ffedcc;
}
.rst-content .important .admonition-title {
background-color: #f0b37e;
}
23 changes: 16 additions & 7 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,23 +11,23 @@
#
# All configuration values have a default; values that are commented out
# serve to show the default.
from __future__ import annotations

import importlib
import os
import re
import sys
from inspect import getmembers, isclass, isfunction
from pathlib import Path
from typing import List, Tuple

from click import secho, style

from kedro import __version__ as release

# -- Project information -----------------------------------------------------

project = "Kedro"
author = "Kedro"
project = "kedro"
author = "kedro"

# The short X.Y version.
version = re.match(r"^([0-9]+\.[0-9]+).*", release).group(1)
Expand Down Expand Up @@ -55,7 +55,7 @@
"notfound.extension",
]

# enable autosummary plugin (table of contents for modules/classes/class
# enable autosummary plugin (table of contents for modules/classes/class
# methods)
autosummary_generate = True
autosummary_generate_overwrite = False
Expand Down Expand Up @@ -102,6 +102,9 @@
"tuple",
"Any",
"Dict",
"dict",
"list",
"set",
"typing.Dict",
"typing.Iterable",
"typing.List",
Expand All @@ -122,7 +125,9 @@
"kedro.framework.context.context.KedroContext",
"kedro.framework.startup.ProjectMetadata",
"abc.ABC",
"Path",
"pathlib.Path",
"PurePosixPath",
"pathlib.PurePosixPath",
"requests.auth.AuthBase",
"google.oauth2.credentials.Credentials",
Expand All @@ -133,6 +138,7 @@
"kedro.extras.datasets.pandas.json_dataset.JSONDataSet",
"kedro_datasets.pandas.json_dataset.JSONDataSet",
"pluggy._manager.PluginManager",
"PluginManager",
"_DI",
"_DO",
# The statements below were added after subclassing UserDict in AbstractConfigLoader.
Expand Down Expand Up @@ -185,7 +191,7 @@
here = Path(__file__).parent.absolute()
html_logo = str(here / "kedro_logo.svg")

# Theme options are theme-specific and customize the look and feel of a theme
# Theme options are theme-specific and customise the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
Expand All @@ -195,6 +201,9 @@
# to ignore or allow certain links.
html_extra_path = [str(here / "robots.txt")]

# Likewise for search console verification
html_extra_path = [str(here / "googlebce3ad2fda582ae8.html")]

# Removes, from all docs, the copyright footer.
html_show_copyright = False

Expand Down Expand Up @@ -329,7 +338,7 @@ def remove_arrows_in_examples(lines):
lines[i] = line.replace(">>>", "")


def autolink_replacements(what: str) -> List[Tuple[str, str, str]]:
def autolink_replacements(what: str) -> list[tuple[str, str, str]]:
"""
Create a list containing replacement tuples of the form:
(``regex``, ``replacement``, ``obj``) for all classes and methods which are
Expand Down Expand Up @@ -401,7 +410,7 @@ def autolink_replacements(what: str) -> List[Tuple[str, str, str]]:
return replacements, suggestions


def log_suggestions(lines: List[str], name: str):
def log_suggestions(lines: list[str], name: str):
"""Use the ``suggestions`` list to log in the terminal places where the
developer has forgotten to surround with double back-ticks class
name/function name references.
Expand Down
26 changes: 24 additions & 2 deletions docs/source/configuration/advanced_configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,7 @@ Although Jinja2 is a very powerful and extremely flexible template engine, which


### How to do templating with the `OmegaConfigLoader`
#### Parameters
Templating or [variable interpolation](https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#variable-interpolation), as it's called in `OmegaConf`, for parameters works out of the box if the template values are within the parameter files or the name of the file that contains the template values follows the same config pattern specified for parameters.
By default, the config pattern for parameters is: `["parameters*", "parameters*/**", "**/parameters*"]`.
Suppose you have one parameters file called `parameters.yml` containing parameters with `omegaconf` placeholders like this:
Expand All @@ -236,10 +237,31 @@ data:

Since both of the file names (`parameters.yml` and `parameters_globals.yml`) match the config pattern for parameters, the `OmegaConfigLoader` will load the files and resolve the placeholders correctly.

```{note}
Templating currently only works for parameter files, but not for catalog files.
#### Catalog
From Kedro `0.18.10` templating also works for catalog files. To enable templating in the catalog you need to ensure that the template values are within the catalog files or the name of the file that contains the template values follows the same config pattern specified for catalogs.
By default, the config pattern for catalogs is: `["catalog*", "catalog*/**", "**/catalog*"]`.

Additionally, any template values in the catalog need to start with an underscore `_`. This is because of how catalog entries are validated. Templated values will neither trigger a key duplication error nor appear in the resulting configuration dictionary.

Suppose you have one catalog file called `catalog.yml` containing entries with `omegaconf` placeholders like this:

```yaml
companies:
type: ${_pandas.type}
filepath: data/01_raw/companies.csv
```

and a file containing the template values called `catalog_globals.yml`:
```yaml
_pandas:
type: pandas.CSVDataSet
```

Since both of the file names (`catalog.yml` and `catalog_globals.yml`) match the config pattern for catalogs, the `OmegaConfigLoader` will load the files and resolve the placeholders correctly.

#### Other configuration files
It's also possible to use variable interpolation in configuration files other than parameters and catalog, such as custom spark or mlflow configuration. This works in the same way as variable interpolation in parameter files. You can still use the underscore for the templated values if you want, but it's not mandatory like it is for catalog files.

### How to use custom resolvers in the `OmegaConfigLoader`
`Omegaconf` provides functionality to [register custom resolvers](https://omegaconf.readthedocs.io/en/2.3_branch/usage.html#resolvers) for templated values. You can use these custom resolves within Kedro by extending the [`OmegaConfigLoader`](/kedro.config.OmegaConfigLoader) class.
The example below illustrates this:
Expand Down
3 changes: 2 additions & 1 deletion docs/source/configuration/configuration_basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,8 @@ Kedro merges configuration information and returns a configuration dictionary ac
* If any two configuration files located inside the **same** environment path (such as `conf/base/`) contain the same top-level key, the configuration loader raises a `ValueError` indicating that duplicates are not allowed.
* If two configuration files contain the same top-level key but are in **different** environment paths (for example, one in `conf/base/`, another in `conf/local/`) then the last loaded path (`conf/local/`) takes precedence as the key value. `ConfigLoader.get` does not raise any errors but a `DEBUG` level log message is emitted with information on the overridden keys.

When using the default `ConfigLoader` or the `TemplatedConfigLoader`, any top-level keys that start with `_` are considered hidden (or reserved) and are ignored. Those keys will neither trigger a key duplication error nor appear in the resulting configuration dictionary. However, you can still use such keys, for example, as [YAML anchors and aliases](https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet#anchors).
When using any of the configuration loaders, any top-level keys that start with `_` are considered hidden (or reserved) and are ignored. Those keys will neither trigger a key duplication error nor appear in the resulting configuration dictionary. However, you can still use such keys, for example, as [YAML anchors and aliases](https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet#anchors)
or [to enable templating in the catalog when using the `OmegaConfigLoader`](advanced_configuration.md#how-to-do-templating-with-the-omegaconfigloader).

### Configuration file names
Configuration files will be matched according to file name and type rules. Suppose the config loader needs to fetch the catalog configuration, it will search according to the following rules:
Expand Down
12 changes: 5 additions & 7 deletions docs/source/contribution/development_for_databricks.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
# How to deploy a development version of Kedro to Databricks
# Contribute changes to Kedro that are tested on Databricks

[Databricks](https://www.databricks.com/) is a cloud-based platform for data engineering and data science that many Kedro users deploy their Kedro projects to. Extending and improving the Kedro experience on Databricks is important to a large part of Kedro's user base.
Many Kedro users deploy their projects to [Databricks](https://www.databricks.com/), a cloud-based platform for data engineering and data science. We encourage contributions to extend and improve the experience for Kedro users on Databricks; this guide explains how to efficiently test your locally modified version of Kedro on Databricks as part of a build-and-test development cycle.

## Introduction

This guide describes how to efficiently develop features and fixes for Kedro on Databricks. Using this guide, you will be able to quickly test your locally modified version of Kedro on Databricks as part of a build-and-test development cycle.
## How to deploy a development version of Kedro to Databricks

```{note}
This page is for people developing changes to Kedro that need to test them on Databricks. If you are working on a Kedro project and need more information about workflows, consult the [documentation for developing a Kedro project on Databricks](../integrations/databricks_workspace.md).
This page is for **contributors** developing changes to Kedro that need to test them on Databricks. If you are a Kedro user working on an individual or team project and need more information about workflows, consult the [documentation pages for developing a Kedro project on Databricks](../deployment/databricks/index.md).
```

## Prerequisites
Expand All @@ -27,7 +25,7 @@ The development workflow for Kedro on Databricks is similar to the one for Kedro

To make developing Kedro for Databricks easier, Kedro comes with a `Makefile` target named `databricks-build` that automates the process of building a wheel file and installing this on your Databricks cluster to save development time.

### Setup the Databricks CLI to test a Kedro build
### How to set up the Databricks CLI to test a Kedro build

Before you use `make databricks-build`, you must [set up the Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html#set-up-the-cli).

Expand Down
Loading

0 comments on commit ab1f7c8

Please sign in to comment.