Skip to content

Commit

Permalink
Update databricks deployment docs (#2404)
Browse files Browse the repository at this point in the history
Signed-off-by: Ahdra Merali <[email protected]>
  • Loading branch information
AhdraMeraliQB authored Mar 14, 2023
1 parent 884f522 commit 63fb4ca
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 6 deletions.
10 changes: 4 additions & 6 deletions docs/source/deployment/databricks.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,22 +60,20 @@ You should get a similar output:
```console
...
[08/09/22 11:23:30] INFO Model has accuracy of 0.933 on test data. nodes.py:74
INFO Saving data to 'metrics' (MetricsDataSet)... data_catalog.py:382
INFO Completed 3 out of 3 tasks sequential_runner.py:85
INFO Pipeline execution completed successfully. runner.py:89
```
### 3. Create a Databricks cluster

If you already have an active cluster with runtime version `7.1`, you can skip this step. Here is [how to find clusters in your Databricks workspace](https://docs.databricks.com/clusters/clusters-manage.html).
If you already have an active cluster with runtime version `7.3`, you can skip this step. Here is [how to find clusters in your Databricks workspace](https://docs.databricks.com/clusters/clusters-manage.html).

Follow the [Databricks official guide to create a new cluster](https://docs.databricks.com/clusters/create-cluster.html). For the purpose of this tutorial (and to minimise costs) we recommend the following settings:
* Runtime: `7.1 (Scala 2.12, Spark 3.0.0)`
* Runtime: `7.3 (Scala 2.12, Spark 3.0.1)`
* Enable autoscaling: `off`
* Terminate after 120 minutes of inactivity: `on`
* Worker type: `m4.large`
* Worker type: `Standard_DS3_v2`
* Driver Type: `Same as worker`
* Workers: `2`
* Advanced options -> Instances -> # Volumes: `1`

While your cluster is being provisioned, you can continue to the next step.

Expand Down Expand Up @@ -213,7 +211,7 @@ from kedro.framework.startup import bootstrap_project

bootstrap_project(project_root)

with KedroSession.create(project_path=project_root) as session:
with KedroSession.create(project_path=project_root, env="databricks") as session:
session.run()
```

Expand Down
Binary file modified docs/source/meta/images/databricks_notebook_example.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 63fb4ca

Please sign in to comment.