Databricks Cost Optimization: Where to Look First

Databricks cost conversations go wrong when they start with a discount and skip the estate. Spend follows architecture: what runs, how often, on which compute, written in which layout, owned by whom.

Look at interactive compute in production

All-purpose clusters are for exploration. Production jobs should run on job compute (or equivalent automated compute), with policies that prevent a notebook from becoming the night-time warehouse. Shared interactive clusters left on overnight are still one of the fastest ways to waste money without producing a data product.

Match ingestion cadence to the decision

Databricks' medallion guidance is direct about cost versus latency. Continuous incremental ingestion (always-on streaming) costs more and reduces latency. Triggered incremental ingestion costs less and accepts higher latency. Batch is cheaper still and slowest.

If a dashboard is refreshed every morning, a continuous pipeline is not “more real-time.” It is an architectural choice you have not priced.

Inspect SQL warehouses and gold models together

A large warehouse on an unmodeled table will stay large. Predicate-unfriendly file layout, missing partitioning or clustering where it is warranted, and SELECT * dashboards are query-design problems. Scaling the warehouse treats the symptom for one meeting.

Name the expensive jobs

• Which jobs consume the most DBUs, and do they still have a consumer?
• Which jobs rewrite the same gold table every run instead of incrementally processing?
• Which pipelines failed into retries that process the world again?
• Which ML or serving endpoints are idle but provisioned?

Attribution before austerity

Teams cannot manage a shared pool. Job names, tags, warehouse owners, and catalog boundaries should make spend attributable to a domain. FinOps that arrives as a monthly spreadsheet, with no architectural owner, produces a week of cluster shrinking and a quiet return to the old pattern.

Do not promise a percentage saving from a blog post. The first output of cost work is a map of drivers. The second is a change to those drivers. Savings are a result, not a claim.

If the platform is expensive because the architecture is duplicated, optimization is a roadmap problem. If it is expensive because a few jobs are badly designed, it is an engineering problem. Those are different engagements. Start by telling them apart.