Lakeflow Architecture for Modern Data Engineering

Lakeflow is Databricks' data engineering umbrella: connecting sources, ingesting, transforming (including Spark Declarative Pipelines), streaming, and orchestrating with Lakeflow Jobs. Using those products well is an architecture problem. The features do not choose grain, SLAs, or what happens when a source schema changes.

Connect and ingest against source behavior

Sources are not equivalent. A SaaS API, an operational database via CDC, object storage, and an event bus impose different completeness, ordering, and late-data characteristics. Ingestion design starts there. Bronze should preserve fidelity—including unexpected fields—so a source change does not destroy history.

Databricks' medallion guidance recommends keeping bronze close to raw, adding provenance metadata, and avoiding aggressive cleanup at the landing layer. That is how you can reprocess.

Transform in silver; serve from gold

Silver is where validation, deduplication, type casting, and the first stable grain belong. Databricks advises building silver from bronze rather than writing silver directly from ingestion, so schema surprises and corrupt records can be isolated.

Gold is the consumption contract: dimensional models, aggregations, and domain-specific products. If every team queries silver with a different join, you do not have a semantic layer. You have a shared staging area.

Orchestrate as if failure is normal

• Jobs have dependencies, retries, and alerts.
• Streaming and batch cadences are explicit.
• Data quality expectations fail the pipeline or quarantine rows—on purpose.
• Lineage through Unity Catalog is a design goal, not a screenshot.

Declarative pipelines are a discipline

Spark Declarative Pipelines (and related Lakeflow pipeline patterns) reduce the amount of custom orchestration you must keep alive. They do not remove the need to define expectations, handle CDC deletes, or decide streaming versus triggered incremental processing.

A modern data engineering architecture on Databricks is still: sources → contracts → layers → jobs → consumers. Lakeflow is how that architecture is implemented. It is not a substitute for writing the architecture down.