DataBricks CLI eXtensions - aka dbx is a CLI tool for advanced Databricks jobs management.
dbx simplifies jobs launch and deployment process across multiple environments.
It also helps to package your project and deliver it to your Databricks environment in a versioned fashion.
Designed in a CLI-first manner, it is built to be actively used both inside CI/CD pipelines and as a part of local tooling for fast prototyping.
- Python Version > 3.6
piporconda
- with
pip:
pip install dbx
Please refer to the Quickstart section.
Please refer to the docs page.
| Tool | Comment |
|---|---|
| databricks-cli | dbx is NOT a replacement for databricks-cli. Quite the opposite - dbx is heavily dependent on databricks-cli and uses most of the APIs exactly from databricks-cli SDK. |
| mlflow cli | dbx is NOT a replacement for mlflow cli. dbx uses some of the MLflow APIs under the hood to store serialized job objects, but doesn't use mlflow CLI directly. |
| Databricks Terraform Provider | While dbx is primarily oriented on versioned job management, Databricks Terraform Provider provides much wider set of infrastructure settings. In comparison, dbx doesn't provide infrastructure management capabilities, but brings more flexible deployment and launch options. |
| Databricks Stack CLI | Databricks Stack CLI is a great component for managing a stack of objects. dbx concentrates on the versioning and packaging jobs together, not treating files and notebooks as a separate component. |
Development:
dbxcurrently doesn't provide interactive debugging capabilities.If you want to use interactive debugging, you can use Databricks Connect +dbxfor deployment operations.dbx executeonly supports Python-based projects which usespark_python_task(Notebooks or Repos are not supported indbx execute).dbx executecan only be used on clusters with Databricks ML Runtime 7.X or higher.
General:
dbxdoesn't support Delta Live Tables at the moment.
For CLI interfaces, we support SemVer approach. However, for API components we don't use SemVer as of now.
This may lead to instability when using dbx API methods directly.
This software is provided as-is and is not officially supported by Databricks through customer technical support channels. Support, questions, and feature requests can be communicated through the Issues page of this repo. Please see the legal agreement and understand that issues with the use of this code will not be answered or investigated by Databricks Support.
Issues with dbx? Found a bug? Have a great idea for an addition? Feel free to file an issue.
Please find more details about contributing to dbx in the contributing doc.