Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. DBUs and Databricks cost

Snowflake, BigQuery & Databricks · Databricks

DBUs and Databricks cost

Mediumwarehouses-39
databricksdbucostcluster-policiesspot

Question

What is a DBU, and how do you control Databricks costs?

Solution

A DBU, or Databricks Unit, is a normalized unit of processing power per hour that Databricks uses for billing. Your Databricks charge is the number of DBUs consumed times a price per DBU. The price depends on the type of compute (job, all-purpose, SQL, serverless) and your plan.

What you pay in total

With classic compute, you pay two bills: Databricks, for DBUs, and your cloud provider for the virtual machines and storage. With serverless, you pay one combined bill, and there are no separate VMs in your cloud account.

classic cost  = (DBUs/hour x price per DBU x hours) + VM cost + storage
serverless    = DBUs consumed x price per DBU   (compute included)

Bigger machines burn more DBUs per hour, so cost grows with node count and size.

Ways to control cost

  • Run production on job compute, not all-purpose clusters, because the DBU rate is lower and clusters end when the job ends.
  • Autoscaling, with sensible minimum and maximum workers, so the cluster shrinks when the load is light.
  • Auto-termination on all interactive compute.
  • Spot (preemptible) instances for workers in fault-tolerant batch jobs, which cuts VM cost a lot. Keep the driver on-demand. Spark can recompute lost work, but a job with long shuffles can slow down if executors are reclaimed often.
  • Cluster policies, which set limits, such as the maximum size, allowed instance types and required tags, so people cannot create very large clusters.
  • Right-sizing, based on the Spark UI: if CPU is at 20 percent and memory is half free, use smaller nodes.
  • Photon: it raises the DBU rate, but it can finish a query faster, so the total can be lower for scan-heavy SQL. Test your workload.

Seeing where the money goes

Databricks provides system tables (for example billing usage tables) that record DBU usage by workspace, job, cluster and tags. Tag clusters and jobs by team and project, and build a dashboard. This gives you the same visibility that query tags give in Snowflake.

Interview angle

Say that cost is a product of configuration choices: compute type, size, runtime and idle time. Then list three or four of the levers above and say that you would measure before and after.

PreviousNext