We’re happy to announce the Common Availability (GA) of help for orchestrating dbt tasks in Databricks Workflows. For the reason that begin of Public Preview, we’ve lots of of consumers leverage this integration with dbt to collaboratively rework, take a look at, and doc knowledge in Databricks SQL warehouses.
With dbt help in Workflows, your dbt mission is retrieved from a Git repository, and a single-node cluster is launched with dbt-core and mission dependencies on it. The SQL generated by dbt is run on a serverless SQL warehouse, offering straightforward debugging and nice efficiency. There are additionally strong and operational capabilities, reminiscent of the power to restore failed runs and ship alerts through Slack or a webhook vacation spot when a dbt process fails, to not point out the power to handle such jobs and retrieve dbt artifacts reminiscent of logs by the Jobs API.
With GA, we’ve prolonged help to SQL Professional Warehouses along with present help for serverless SQL Warehouses. Furthermore, we’re completely satisfied to announce help for Databricks on Google Cloud Platform (GCP). Lineage from transforms laid out in dbt tasks can also be routinely captured in Unity Catalog. Lastly, much more dbt group packages reminiscent of dbt-artifacts now work with Databricks.
To get began with dbt on Databricks, merely run “pip set up dbt-databricks.” This installs the open supply dbt-databricks package deal constructed along with dbt Labs and different contributors. You may observe our detailed information to get began with an instance mission. When you commit your supply code to a git repository, you should utilize Databricks Workflows to execute your dbt fashions in manufacturing (see our docs for (AWS | Azure | GCP).
