Common Databricks Architecture Mistakes

Most Databricks estates that feel “stuck” are not stuck on Spark. They are stuck on decisions that were never made. These are the patterns we see when a platform is present and a system is not.

1. No environment strategy

Development and production share a workspace, a catalog, or both. Change control is social. A test job can write to a table an executive dashboard reads.

2. Unity Catalog as a lift of folders

Workspace objects were copied into catalogs without a domain model. Privileges were copied onto users. Ownership stayed with whoever ran the notebook. The catalog exists; governance does not.

3. Medallion as interior decoration

bronze / silver / gold appear in names. Quality tests, reprocessing, and consumers do not. Downstream jobs still read whatever table was updated last.

4. Notebooks as the production runtime

Interactive clusters, hidden widgets, and undocumented run order. When the author is on leave, the job is folklore.

5. Every workload on the same compute class

BI, ETL, and exploration compete. Cost cannot be attributed. Performance work becomes “make the cluster bigger.”

6. Semantics living only in BI

The metric layer cannot be reused by SQL, applications, or agents. The next team rebuilds revenue.

7. AI as a parallel estate

Vector indexes and prompt apps sit outside Unity Catalog, with a service account that can see everything. The lakehouse did not get an AI strategy. It got a shadow copy.

8. Migration without retirement

The old warehouse still runs. Databricks runs. Nobody knows which one is allowed to be true. You are paying for indecision.

None of these require a new product. They require architecture, ownership, and a production standard. That is slower to announce than a feature rollout, and it is the work that makes the platform usable.