Databricks
Run SQL statements on a Databricks SQL warehouse and trigger jobs from Tessel. Start workflows when a job run finishes and write results back to a staging table.
Warehouse
Category
Batch
Sync mode
p95 3.8 s
Latency
8 min
Setup time
What it can do
- Job run succeededEvent
- Job run failedEvent
- Scheduled queryPoll
- Run a SQL queryQuery
- Read query resultsRead
- Trigger a jobWrite
- Write to a staging tableBatch
Overview
The Databricks connector runs SQL through the Statement Execution API and starts jobs through the Jobs API. Results come back as typed rows with the column schema attached, and every query in a run shows its statement ID and duration in the trace.
Data teams use it to start a reconciliation workflow when a nightly job finishes, to page on-call in Slack when a job run fails and to load scored rows from Kafka-fed tables into Postgres.
Supported workspaces
Databricks on AWS, Azure and Google Cloud
Serverless and pro SQL warehouses
Unity Catalog tables, with three-part names in queries
How queries work
Tessel submits each statement to the warehouse you pick and waits up to 30 seconds inline. Longer queries continue in the background and Tessel polls for the result. If the warehouse is stopped, the first query starts it and the trace shows the wait.
Large results
Results over 25 MB are fetched as chunks through external links and streamed into the next step. Write-backs go through MERGE on a key you choose, so a retried batch never writes rows twice.
Permissions needed
Create a service principal for Tessel and connect with OAuth machine-to-machine. It needs use access to one SQL warehouse and read access to the tables you query. Write access is only required for the staging schema.
CAN USEon the SQL warehouseUSE CATALOG,USE SCHEMAandSELECTon source tablesMODIFYon the staging schema andCAN MANAGE RUNon jobs you trigger
Six steps. About eight minutes.
Most of the time goes into Unity Catalog grants. Tessel runs a test query before you finish.
- Create a service principal and an OAuth secretDatabricks2 min
- Grant warehouse and Unity Catalog permissionsDatabricks2 min
- Add Tessel's webhook as a notification destinationDatabricks1 min
- Allow Tessel's static IPs if IP access lists are onNetwork1 min
- Paste the workspace URL, client ID and warehouse IDTessel2 min
- Run a test query and check the result rowsVerify3.8 s
Your lakehouse, in a workflow today.
Connect Databricks on the free plan. Up to three builders and 10,000 runs a month, no card required.