The Databricks Migration Harness

The trust gap in agentic systems

Every data platform vendor is shipping AI agents now – agents that write ETL code, generate infrastructure, and make architectural decisions that used to take a team of engineers weeks to work through. The demos are genuinely impressive. An agent reads a legacy schema and produces a working pipeline in minutes.

But the demo is never the hard part. The hard part is the question that comes right after: how do I know this is actually correct?

That question is where most enterprise AI adoption quietly stalls. Not on capability – on trust. A generated pipeline that’s 95% right is, in a production data platform, not usably right. And “the model seemed confident” is not a verification strategy.

This is the problem we ran into building what we now call the Databricks Migration Harness – the quality and verification layer underneath our agent-driven migration pipeline for moving legacy analytics platforms onto Databricks. This post is about what we built, what we got right, what we got wrong, and where we think this goes next – including a fairly candid self-audit we did before writing any of this down, because we didn’t want to publish claims we hadn’t actually checked.

What we mean by “harness engineering”

A quick note on vocabulary, because the term we’re using has a specific and fairly recent history, and using it carelessly would be its own kind of dishonesty.

In February 2026, Mitchell Hashimoto – co-founder of HashiCorp and creator of Terraform – published a widely-read post describing a habit he’d developed while building with AI coding agents: every time an agent made a mistake, he engineered a permanent fix into the agent’s environment rather than just re-prompting. He called this “engineering the harness.” The broader AI engineering community – including organizations such as Anthropic, OpenAI, Martin Fowler, and Databricks – has increasingly emphasized the same idea: that successful AI systems depend not only on the model, but on the surrounding engineering infrastructure of tools, context, permissions, evaluation, and feedback. A useful way to think about modern agent systems is: Agent = Model + Harness. The model is the raw reasoning capability. The harness is everything else – the tools it can call, the memory and context it operates with, the permissions that bound what it’s allowed to do, and the verification and feedback loops that catch it when it’s wrong.

Worth being precise here, because “harness” describes what the harness actually does to each agent: it lets agents do a particular thing in a particular way. Concretely, that comes down to three things. Boundaries – the harness controls what information flows into each phase before it runs, so an agent can’t act outside what it’s been given. When Pipeline Forge runs, the Architecture phase’s approved design is already pre-loaded into its execution state; it can build what that design defines and nothing else, not because a rule tells it not to, but because anything else simply isn’t there to work with. Memory – workflow state within a phase (pausing at a checkpoint and resuming exactly where it left off) and artifact handoff between phases (one phase’s certified output becoming the next phase’s input) are both forms of memory sharing across agents. Within the normal harness workflow, this inheritance is automatic; if a phase is triggered independently of the pipeline, it has to explicitly request the prerequisite information rather than receiving it for free. Tools – each phase also has its own execution tools: schema mappers, DDL generators, a transformation parser, the pipeline-package generator. Tools are how the work actually gets done, but they only operate on what boundaries and memory make available to them.

Most of the public writing on agent harnesses focuses on tools, memory, and context at the level of a single agent. Verification and permissions, and what happens once you chain several agent-harnesses together into a pipeline, get comparatively little attention. That imbalance is exactly what this post is about. We built the verification piece first, for reasons specific to enterprise data migration, and it taught us things about the rest of the harness that we think are worth sharing.

A harness, in practice, is more than a checkpoint – it governs an agent’s full cycle of work through a Plan → Execute → Verify pattern common in agentic systems: what the agent is allowed to attempt, the boundaries it operates within, and the verification that catches it when it’s wrong.

Within that loop, a gate is the verification piece specifically. Each gate verifies a contract: a formal specification of what the previous phase must have produced before the next phase is permitted to begin. In our world, a gate produces a structured approval decision – deterministic where the check is automated, explicit and auditable where human judgment is required. Not a developer test. Not an internal eval. Something the client sees, can configure, and signs off on.

The Databricks Migration Harness: what it actually does

Migrating a legacy analytics platform onto Databricks is high-stakes, low-tolerance-for-error work. Get a schema mapping wrong and a finance report is silently incorrect for a quarter before anyone notices. Unlike a conversational AI response, which can simply be regenerated if it’s wrong, a migration artifact becomes an input to the next phase – errors compound forward instead of disappearing when you ask again. So our migration pipeline is built around six phases, each producing an artifact, each gated by a verification checkpoint before the next phase can begin.

Rather than building one monolithic evaluator, we partitioned verification into six independent harnesses aligned to each migration phase. Each harness owns a narrowly scoped contract, which means failures are isolated early, reviewed independently, and can be improved without affecting unrelated stages of the pipeline. A schema validation failure in Discovery doesn’t contaminate the Architecture gate’s verdict – and the client sees exactly which phase, which contract, and which rule failed.

Each harness is intentionally more than an evaluation stage. It encapsulates the execution context for a migration phase – its tools, state, verification rules, approval workflow, audit history, and feedback mechanisms. Evaluation is one responsibility of the harness, not the harness itself. And the reason these are six separate harnesses rather than six functions inside one orchestrator is architectural, not incidental: each phase operates with a different execution environment, toolset, and success criteria, because each migration phase requires a different form of reasoning and verification. Rather than six isolated checkpoints, the phases form a chain of trust – each harness certifies that its phase’s contract has been satisfied before handing a verified artifact to the next harness in line.

• Discovery – scans the source system and produces a full inventory: tables, schemas, volumes, dependencies. Gate: compares discovered assets against expected inventory and identifies potential gaps requiring review.
• Architecture – designs the target Bronze/Silver/Gold structure (the medallion architecture pattern Databricks popularized: raw data lands in Bronze, gets cleaned and conformed in Silver, and is aggregated into business-ready tables in Gold). Gate: does the design hold up, and does the client sign off on it before anything gets built?
• Wave Planning – sequences which tables and workloads migrate in which order. Gate: is the sequencing safe – no orphaned dependencies, no broken references mid-wave?
• Pipeline Forge – the most generative phase, producing not just notebook code but ETL transpilation, workflow orchestration, deployment bundles, and CI/CD configuration in a single pass. Gate: does the generated output meet the target platform’s structural requirements and match the approved architecture? This is a second, explicit client sign-off point.
• Deploy – pushes validated artifacts through version control into the target environment.
• Validation – the deepest gate: does the migrated data actually match the source?

That last gate is worth walking through in detail, because it’s the clearest example of how we think about verification.

Reconciliation as layered verification

The naive approach to proving migrated data is correct is a full table diff – compare every row, every column, source versus target. It’s slow, it’s fragile against any legitimate transformation, and it doesn’t tell you why something’s wrong, only that something is.

We layer it instead. The layered approach aligns with what Databricks recommends for migration validation – row counts, aggregate checks, hash comparisons, and lineage – though the specific sequence and implementation below is ours.

• Row counts – do Bronze, Silver, and Gold each have the row counts we’d expect, layer over layer?
• Aggregate metrics – do sums, counts, and null rates per column match, at a coarser grain than row-by-row?
• Hash-based row matching – for anything that fails the aggregate check, hash rows on both sides to efficiently identify candidate mismatches, which are then inspected to isolate the precise column-level differences.
• Schema and type validation – did column types survive the migration, or did something silently coerce?

Each layer is cheap and gives an actionable failure – a hash mismatch pointing to a candidate row range, a schema coercion on a DECIMAL type – something a human or an agent can act on. A full diff that says “12,000 rows don’t match” is not. Results write to a queryable table, so every failure is auditable and can be revisited months later, not just flagged in a Slack message and forgotten.

Checking our own claim against the code

Before publishing any of this, we asked ourselves a harder question than “does our harness work” – we asked “does our architecture actually satisfy the harness definition we just described, or are we using a current-sounding word for something narrower?” We went and checked, against the real implementation, not the pitch deck.

• Verification is solid. The six gates are genuine: deterministic, logged, and they block progression on failure. This is the strongest and most complete part of what we’ve built.
• Memory is real, but scoped. Different forms of memory operate at different scopes: workflow execution state within a phase, which genuinely pauses at a human-review checkpoint and resumes exactly where it left off; and persisted artifacts between phases, where one phase’s completed output feeds the next phase as input. Both of these are real and working today. A third form – accumulated failure knowledge feeding back into future runs – is live on Pipeline Forge and being extended to the rest of the gates; see the next section for exactly where that stands.
• Permissions are more enforced than a simple “prompt-based guidance” label would suggest. Two mechanisms are real and code-level, not just instructions the model is asked to follow: status checks block a phase from starting before the prior phase is actually complete, and – as the harness blueprint above describes – a phase can only act on data that’s been loaded into its execution context. If a phase is triggered out of the normal sequence, it doesn’t silently proceed; it asks for the prerequisite information before doing anything. Together, these close off two of the most consequential failure modes: acting out of order, and acting on data that was never approved. What’s still mostly prompt-based is fine-grained tool-level access control – restricting exactly which actions an agent can take within a phase, beyond what its state already scopes it to. The one exception there is a code-level guard that blocks destructive SQL before it executes, regardless of what the model intended. We’re extending that same pattern – enforcement built into the platform rather than requested of the model – to tool-level access more broadly.

We’re including this audit because a harness that claims every component is already solved isn’t more trustworthy than an agent with no harness at all – it’s just a different kind of overclaim. The whole argument of this post is that verification earns trust by forcing honesty about what’s actually built. It would have been strange to make that argument and then not apply it to ourselves.

Making harnesses self-improving

A verification gate that only says pass/fail is useful. A verification gate that makes the next run better is more useful – and on Pipeline Forge, the most generative and error-prone of our six phases, we’ve already built it.

Here’s the high-level design. When the Pipeline Forge gate fails a generated artifact, the failure isn’t just logged as a raw error – it’s classified into a taxonomy: a type-coercion issue, a naming-convention violation, a missing cast, a structural mismatch against the approved architecture. Each classified failure is written to a structured, queryable store alongside the context it occurred in: what kind of source table, what target pattern, what rule it broke. Before the next generation run – the next table, the next wave – the harness pulls the failures most relevant to what it’s about to generate and folds them into the generation context as explicit constraints: this source type has previously failed on X, avoid it; always do Y instead. Over successive waves, the same class of failure shows up less, because the harness is actively feeding its own history back into the next attempt rather than starting cold every time.

That’s the building block: classify → store → inject. It’s proven out concretely on Pipeline Forge today. What’s still in progress is extending the same mechanism to the other five gates – Discovery, Architecture, Wave Planning, Deploy, and Validation – each of which will need its own failure taxonomy, since a schema-discovery failure and a generated-code failure aren’t the same shape of thing. The building block is built; generalizing it across the harness is the work ahead.

The more mature version of this goes further: failures feeding a structured knowledge base that eventually informs planning and evaluation criteria across gates, not just the immediate next generation attempt within one gate. That’s further out than what’s running today, but it’s the direction.

Worth being explicit about what’s actually improving here: the objective isn’t to make the language model smarter. The objective is to make the migration system more reliable by continuously enriching the harness around the model. That distinction is close to the philosophical core of what harness engineering is trying to do – Pipeline Forge doesn’t get better because the underlying model changed; it gets better because the harness around it accumulates and reuses its own history.

A few things we’ve learned building this on Pipeline Forge that matter for extending it well: failures have to be structured, not free-text, or there’s nothing for the next run to actually use; the failure output has to be shaped so it can go straight into the generation context, not require a human to translate it first; and feedback has to arrive before the next run, not in a retro two weeks later. Human override always stays available – self-improving doesn’t mean fully autonomous.

It’s also worth connecting this back to permissions: sequencing and data-boundary enforcement – the mechanisms described above – already prevent an agent from acting out of order or on data it hasn’t been given. But a harness that’s teaching an agent to act more independently over time needs one more layer to stay safe as that independence grows: enforced walls at the tool level, not just the boundaries that today’s state-scoping already provides. Self-improvement and enforced tool-level permissions aren’t separate workstreams; the second is what keeps the first safe as it matures.

Beyond migration: extending the harness to everyday operations

Migration is a bounded, one-time problem – move these tables, from this system, to Databricks, once. But the day after a migration finishes, a different set of problems starts: enhancements, bug fixes, schema drift, performance tuning. The same verification pattern applies there too, just on a different cadence.

• Enhancements – a new column or transformation goes through the same reconciliation-style check: did this change break any existing downstream contract?
• Bug fixes – an agent proposes a fix for a data quality issue; the harness verifies the fix resolves the problem without regressing anything else.
• Schema evolution – a source system adds a column; the harness checks whether the target schema can absorb the change safely.
• Drift detection – quality degrades slowly over time; the same layered checks that validate a migration can catch drift before it compounds.

The difference is cadence, not architecture. Migration harnesses gate discrete project phases. Operational harnesses run continuously, on every change. The underlying verification logic – row counts, aggregates, hashes, schema checks – doesn’t change; what changes is that it becomes permanent infrastructure instead of a project deliverable that gets retired at go-live.

Platform agnosticism: what’s reusable beyond Databricks

A fair question: how much of this is actually Databricks-specific, versus a pattern that would work anywhere?

Some of it is genuinely tied to the platform – validating generated pipeline code against Databricks’ specific syntax, or checking against Unity Catalog’s governance model. But the core verification pattern isn’t platform-specific at all. Layered reconciliation – row counts, aggregates, hashes, schema – works against Snowflake, BigQuery, or any SQL-speaking warehouse. A structured failure taxonomy doesn’t care what platform produced the failure. The feedback loop design is agent-agnostic. Customer sign-off gates apply to any high-stakes agent-generated infrastructure, not just data migration.

The practical path to platform-agnosticism is pulling platform-specific logic out into pluggable adapters – validation, generation, and governance – each speaking that platform’s dialect, while the core verification engine, the failure taxonomy, the audit trail, and the customer-facing experience stay the same underneath. Most of what makes the harness valuable was never actually about Databricks in the first place.

Where we go from here

In the near term: harden what exists for production scale, extend the self-improving feedback loop already running on Pipeline Forge to the remaining five gates, and extend enforced, platform-level access control to the tool layer the way sequencing and data boundaries are already enforced today. In the medium term: prove the pluggable-adapter architecture with a second platform target, and extend the harness beyond migration projects into standing operational infrastructure for teams running their own agent-assisted pipelines.

The longer-term shift we think is coming: today, companies adopt agents first and then struggle to build trust and control around them after the fact. We think that inverts. Harnesses become the foundational infrastructure companies adopt first, and agents get built to operate within the constraints that infrastructure defines – rather than harnesses being a safety net bolted on after an agent is already running loose in production.

Three closing thoughts

Harnesses may end up mattering more than the agents they wrap. The agent is the visible part – it writes the code, drafts the design, makes the call. The harness is unglamorous by comparison: it validates, it gates, it says no. But the harness is where trust actually gets built. We suspect the skill of building good verification and feedback loops around an agent will end up being worth more than the skill of prompting the agent in the first place.

Self-improvement through verification is a different kind of learning than bigger models. The industry default has been to make models larger and more broadly capable. A model constrained by a good harness has a different path available: improving on a specific domain – data migration to a specific target platform, say – by learning what that domain’s rules actually are, without ever being retrained. We’ve built this for one gate – Pipeline Forge classifies, stores, and reuses its own failures already – and extending the same mechanism across the rest of the harness is the work ahead. Either way, it’s the direction we think is more interesting than a bigger model: the work isn’t making the model smarter, it’s making the harness around it sharper.

Harnesses are what make the interface between humans and agents legible. Right now, in most agentic systems, a human eyeballs an output and decides whether to trust it – an ad hoc, unrepeatable judgment call. A mature harness turns that into something structured: the agent produces output, the harness checks it against an explicit, inspectable specification, and the human signs off on a verification result rather than a vibe. That’s roughly the same shift software engineering went through decades ago, moving from “someone reviewed the code and it looked fine” to automated test suites. We think agentic systems are early in that same transition, and the harness is the part of the stack doing the work.

As models continue to improve, we expect competitive advantage to shift increasingly toward the engineering systems that govern how those models are applied – the tightest, most honest feedback loop between what an agent produces, what a harness verifies, and what an agent learns from that verification the next time around. And that includes being willing to publish an honest account of exactly how far along that loop actually is, rather than how far along it sounds.