The Problem Nobody Talks AboutThe Problem Nobody Talks About

Data warehouses are built by engineers. Dashboards are built by analysts. Between them lies a chasm — the semantic layer — and it’s almost always hand-crafted, poorly documented, and perpetually out of date.

Consider what building a semantic layer actually requires today. A data engineer must deeply understand the normalised schema: every foreign key, every relationship chain, every edge case where a left join becomes an inner join. They must then interview stakeholders to understand the analytical questions the business needs answered. Only then can they begin designing wide tables — denormalised, pre-joined views that flatten a complex relational model into something a BI tool can query without six-table joins.

This process typically takes days to weeks for a mid-complexity schema. And the result? A static artefact that starts decaying the moment the schema changes or a new business question emerges.

The Core Tension

Normalised schemas optimise for data integrity. Analytical queries optimise for speed and simplicity. The semantic layer is the translation layer between these two worlds — and it has been painfully manual for decades.

The downstream costs are significant. When the semantic layer is missing or poorly designed, analysts write ad-hoc queries with redundant joins, dashboards slow to a crawl, Text2SQL tools hallucinate because they can’t navigate complex schemas, and every new business question triggers another round of engineering tickets.

Enlarge

Enlarge

Enlarge

Enlarge

The Business Case

This isn’t just a technical inconvenience — it’s a strategic bottleneck. Organisations that can’t translate raw data into queryable, governed, analyst-friendly tables cannot move at the speed of their own questions.

Enlarge

A well-designed semantic layer delivers three things simultaneously: it makes analysts self-sufficient (no more waiting on engineering for every new slice of data), it makes queries dramatically faster (pre-joined views eliminate runtime join costs), and it makes Text2SQL systems reliable (flat tables with descriptive column names are exactly what language models need to generate accurate SQL).

The question was never whether organisations need a semantic layer. It was whether the process of building one could be automated without sacrificing quality. That’s the question this tool answers.

Why Now? The Convergence Moment

Three capabilities had to mature simultaneously to make this possible.

1. LLMs That Understand Schema Semantics

Modern large language models don’t just parse JSON — they understand what a DEPARTMENT_ID foreign key means, how it relates to an EMPLOYEES table, and why an analyst asking about “salary distribution by role band” needs a join path through SALARIES → EMPLOYEES → ROLES. This semantic reasoning is the foundation the entire pipeline rests on.

2. Agentic Orchestration Patterns

A single LLM call can’t do this well. You need a multi-phase pipeline where each stage’s output feeds the next — analysis, optimisation, design, generation, reporting — with deterministic optimisation steps (pure set-logic pruning) interleaved with LLM-powered reasoning. This is agentic AI in its most productive form: not a chatbot, but an autonomous system that completes a complex engineering workflow end-to-end.

The term “agentic data engineering” describes this shift: AI agents that autonomously perform data engineering tasks through structured interfaces, with human engineers reviewing, approving, and governing the output. It is the same architectural pattern as agentic analytics, applied one layer down in the stack.

And the two layers are more connected than the industry typically acknowledges.

What is agentic data engineering?

Agentic data engineering is the practice of using AI agents to autonomously perform data engineering tasks (building pipelines, writing transformations, managing schemas, monitoring data quality, and maintaining documentation) through code-native interfaces with human oversight.

The “agentic” part matters. The agent goes well beyond autocompleting SQL. It can:

Accept a high-level goal (“create a pipeline that ingests Stripe subscription events into our warehouse, deduplicated by event ID, partitioned by date”) Decompose the goal into steps Generate the pipeline definition, transformation logic, and tests Validate against existing schema and conventions Submit the result for review The human engineer reviews, tests, and approves, the same workflow as reviewing a pull request from a junior engineer, except the junior engineer runs at machine speed and never forgets to write tests (if told to).

Enlarge

What can AI agents do in data engineering today?

The capabilities are real and growing. Here is what works in production today, verified beyond demos.

Pipeline generation

Agents generate pipeline definitions (Airflow DAGs, Dagster jobs, dbt models) from natural language descriptions or schema specifications. The output is code: reviewable, testable, version-controlled.

Transformation authoring

Given a source schema and a target requirement, agents write SQL transformations, dbt models, or other transformation logic. They handle joins, type casting, deduplication, and incremental load patterns.

Schema migration and evolution

When source schemas change (a new column appears, a type changes, a table is deprecated), agents analyze the impact downstream, generate migration scripts, and update affected transformations.

Data quality monitoring

Agents generate data quality tests (row counts, null checks, distribution anomalies, freshness checks) based on observed data patterns. They can monitor continuously and flag deviations.

The emerging tool ecosystem

Several major platforms are building agentic data engineering capabilities.

Google BigQuery, Data Engineering Agent

Google’s Gemini-powered agent generates SQL transformations, creates pipelines, and manages BigQuery datasets through natural language. Integrated into the Google Cloud console. Strongest for teams already on BigQuery.

Databricks, AI-powered data engineering

Databricks integrates AI across its lakehouse platform. Agents assist with notebook development, pipeline creation, and data quality. Unity Catalog provides the governance layer. Strong for organizations with a Databricks-centric stack.

Two paths, one index

Every RAG system has two halves that run on very different schedules:

• Offline indexing turns source documents into searchable chunks. It is a batch or streaming data pipeline and should be engineered like one.
• Online querying turns a user question into a grounded answer within a latency budget, typically a few seconds end to end.

Enlarge

The offline path

Parse and clean

PDFs, wikis and tickets all need different parsers. Strip navigation chrome, keep headings and tables, and record where every piece of text came from — you will need that provenance for citations.

Chunk with structure

Fixed-size chunks are a fine baseline, but chunking along document structure (sections, list items, table rows) usually retrieves better. Store metadata such as title, section path, timestamp and access-control labels with every chunk.

Embed and index

Index each chunk twice: as a dense vector for semantic similarity, and as text for keyword search. Hybrid retrieval rescues the queries that embeddings handle poorly — product codes, error messages and acronyms.

The online path

def answer(question: str) -> Answer:
    query = rewrite(question)                       # expand acronyms, fix typos
    candidates = index.hybrid_search(
        query, top_k=40, filters=user_acl_filter()  # never retrieve what the user can't see
    )
    passages = reranker.rerank(question, candidates)[:6]
    prompt = build_prompt(question, passages)
    response = llm.generate(prompt, temperature=0)
    return Answer(text=response.text, citations=[p.source for p in passages])

Apply access-control filters at retrieval time. Filtering the final answer is too late — the model has already seen the restricted text.

Evaluate retrieval and generation separately

When an answer is wrong, you need to know whether the right passage was never retrieved or was retrieved and then ignored.

Build a small “golden set” of 50–100 real questions with known source passages before tuning anything. Without it, every change is a guess.

Wrapping up

Treat the index as a data product with owners, freshness SLAs and quality checks, and treat the LLM as one component of a pipeline you can measure. Do that, and RAG stops being a demo and starts being a system.

What idempotency means for data

A task is idempotent when run(x) followed by run(x) leaves the target in exactly the same state as a single run(x). For data pipelines that implies:

1. No duplicates when a run is retried.
2. No gaps when a run is skipped and later backfilled.
3. Deterministic output for a given input and run parameter.

Enlarge

Pattern 1 — replace a whole partition

When each run owns a well-defined slice of data (usually a date), overwrite that slice atomically instead of appending to it:

(
    daily_events
    .write
    .format("delta")
    .mode("overwrite")
    .option("replaceWhere", f"event_date = '{run_date}'")
    .saveAsTable("silver.events")
)

Running this twice for the same run_date replaces the partition twice — the end state is identical.

Pattern 2 — merge on a natural key

For entity tables that receive updates, upsert on a business key and only apply a change if it is newer than what is already stored:

MERGE INTO silver.customers AS t
USING staged_customers AS s
  ON t.customer_id = s.customer_id
WHEN MATCHED AND s.updated_at > t.updated_at THEN
  UPDATE SET *
WHEN NOT MATCHED THEN
  INSERT *;

UPDATE SET * / INSERT * is Delta Lake syntax. Other engines need explicit column lists, but the pattern is identical.

Pattern 3 — derive time from the run, not the clock

current_date() inside a transformation makes the output depend on when the job ran. Pass the logical run date from the orchestrator instead, and use it everywhere: in filters, partition values and output paths.

Wrapping up

Idempotency is cheap to design in and expensive to retrofit. Pick a write pattern per table, parameterise everything by the logical run date, and retries become the most boring part of your on-call rotation.

Why layer a lakehouse at all?

Most data problems are really trust problems. When a dashboard number looks wrong, the team needs to answer three questions quickly: what did the source actually send, what did we do to it, and which business rule produced the final figure? Layering separates those concerns so each one can be inspected on its own.

Enlarge

The three layers

Bronze — raw and append-only

Bronze is a faithful, replayable copy of what the source sent, plus ingestion metadata such as load time, source file and batch id. Resist the urge to clean anything here: if a transformation bug is found next quarter, bronze is what lets you rebuild everything downstream.

Silver — cleaned and conformed

Silver is where data becomes usable: types are enforced, duplicates removed, late-arriving records merged and entities conformed to shared keys. Silver tables are modelled around business entities (customers, orders, devices), not around source systems.

Gold — business-ready

Gold tables answer specific questions: aggregates, dimensional marts, feature tables and governed metrics. They are small, documented and owned by a team that can explain every column.

Keep bronze retention long and cheap (object storage is inexpensive), but keep gold small and well-documented. Most cost and confusion comes from sprawling, undocumented gold tables.

A gold table, end to end

Gold transformations should be boring, declarative SQL over silver:

CREATE OR REPLACE TABLE gold.daily_revenue AS
SELECT
  order_date,
  region,
  SUM(net_amount)              AS revenue,
  COUNT(DISTINCT customer_id)  AS customers
FROM silver.orders
WHERE status = 'COMPLETED'
GROUP BY order_date, region;

If a gold query needs to reach back into bronze, that is a signal that a silver table is missing.

Never point BI tools at bronze. Raw data changes shape without warning, and dashboards built on it become the first thing to break.

Common anti-patterns

• Source-shaped silver. Copying each source table into silver one-to-one just moves the mess one layer down.
• Business logic in bronze. Filtering or “fixing” records on ingestion destroys your ability to replay.
• Gold for everything. Not every query deserves a persisted table; views over silver are often enough.
• Non-idempotent loads. Re-running a job should never duplicate data — see Idempotent Data Pipelines.

Wrapping up

The medallion architecture is less about the names of the layers and more about the guarantees each layer gives. Write those guarantees down, enforce them with tests, and the lakehouse stays explainable as it grows.