At a recent conference, I was talking about Prophecy’s AI-generated pipelines and I got the question, “...but how do you know it’s right?” It’s a great question, since AI is probabilistic and known to hallucinate–not something you want in your critical data process. It might look good, but underneath there is one small mistake in the business logic that will propagate downstream.
This is a pipeline that's “almost right.” It runs perfectly, then feeds the wrong numbers into your quarterly board deck. It lets you continue making seemingly data-backed decisions, unaware that you're heading in the wrong direction.
AI lets analysts build data pipelines in minutes, which saves tons of time and coordination. But when AI writes the logic, and no one rebuilds the verification an engineer would have added, the pipeline runs clean and fills a dashboard, potentially using the wrong logic or data sets. Nobody notices, because nothing looks broken, hence “almost right.”
That silence is what makes almost right the expensive kind of wrong. Let’s take a look at the potential consequences of this for your business.
What happens downstream
An "almost right" pipeline looks identical to a working one. It runs without errors, the SQL reads cleanly, and the numbers fall within a plausible range. Every check the team relies on clears it, even as the wrong logic runs underneath.
The pipeline runs without error
A query that runs cleanly can still return the wrong answer. On the BIRD benchmark, the best fine-tuned version of LLaMA-3.1 70B produces valid SQL 98.50% of the time, but the correct answer only 68.51% of the time. In other words, nearly a third of clean-running queries came back wrong, even though every one of them was executed without error.
Code review approves it
Review catches errors that look like errors, a syntax mistake, or an obvious bug. The errors an "almost right" pipeline produces don't look like that.
Take a churn report where the AI joins customer records to support tickets by email instead of by customer ID. Anyone with two emails gets counted twice. Churn ticks up a couple of points, the SQL still reads cleanly end to end, and the reviewer sees plausible code and a plausible number. So they sign off. Under a heavy review queue, that fluency feeds a known over-trust effect.
The more reasonable the code looks, the more likely the error survives.
The dashboard looks right
The last check is a person reading the output, and a number that looks right clears it. A revenue figure that runs a few points high still reads like a revenue figure, so finance signs off, and it lands in the board deck. A forecast gets built on the number before anyone reconciles it against the source.
A wrong number that clears every check still reaches a decision, and the cost is that decision plus every one built on top of it before anyone traces the error back.
AI skips the verification that an engineer would build in
When an engineer builds a data pipeline, half the job is writing the logic. The other half is wiring in the verification that catches a wrong result before anyone acts on the data.
The scaffolding an engineer builds in is what would have caught the error early:
- Schema validation and data contracts catch a schema change or a broken service-level agreement (SLA) the moment the data shifts
- Data quality checks stop bad data from spreading, the kind of automated validation that tests accuracy, completeness, validity, consistency, freshness, and uniqueness before anything reaches production
Why just reviewing the output isn't enough
Reviewing the AI's output isn't enough because the analyst who knows the business often isn't the one fluent in SQL, and the engineer fluent in SQL isn't always close enough to the business to catch a number that's subtly off.
Analysts generate these pipelines in minutes. A reviewer who traces every join, filter, and assumption on each one becomes the new bottleneck, the exact delay AI was brought in to remove. So the review gets quicker and thinner under that pressure.
Handing the check back to the AI doesn't help either. When people in one study of AI persuasion pushed back on a model's answer, the model defended its first answer more forcefully the harder they pushed. A tool that confidently justifies wrong work can't be the thing that catches wrong work.
The fix sits earlier in the process. Verification runs as the pipeline is built, and the logic stays in plain view, with the joins, filters, and assumptions a person can actually inspect.
What generate-then-verify looks like for a sales forecasting pipeline
Picture a sales ops analyst building a forecast of next quarter's pipeline coverage by region. The source data lives across a CRM, a billing system, and a spreadsheet of quota assignments, and the deadline is tomorrow's revenue review.
In Prophecy, the analyst catches the bad join before the forecast reaches the CRO. An AI agent drafts the visual data workflow that joins opportunities to accounts, rolls them up by region, and projects them against quota. Each step renders as a visual block with a sample of the rows flowing through it, so the analyst can inspect the logic without reading SQL.
Because the logic stays visible, the analyst spots two problems:
- The opportunity-to-account join sits at the wrong grain, double-counting multi-product deals in enterprise regions
- Closed-lost opportunities are still sitting in the dataset, inflating the pipeline number
From there, the fix is quick: the analyst corrects the join, adds a filter to drop the closed-lost rows, and the regional totals snap back to numbers the regional VPs already recognize.
The safeguards don't stop at build time. When the workflow deploys, it lands on the platform the data team already governs, under the same controls and access rules. Agentic data preparation then wires in the checks an engineer would have added, including:
- Schema validation that flags the moment the CRM team renames a field upstream
- Data quality checks that catch the week a billing extract arrives half-empty
These checks surface the problem before bad data reaches the forecast that the CRO is about to present.
That's the tradeoff worth getting right. Generating a pipeline has become cheap, which is good news for anyone stuck in a request backlog. But an "almost right" pipeline that confidently ships wrong numbers costs more than one that never ran, and the bill grows the longer the error goes unnoticed.
Build the verification back in, and you get the speed without the silent failure. Book a demo to see how Prophecy makes it work.
Try the generate-then-verify approach
Build your first pipeline with AI on the Prophecy Starter Edition for free. Sign up for an account here.
