Routine data workflow changes can stall in engineering queues because demand for analytics outstrips available supply. The constraint isn't the analyst's ability to define the business logic. It's the engineering capacity required to translate each request into production-ready work, validate it, and deploy it safely.
Agentic data preparation can close that gap. But implementation determines whether a team gets speed with trust or speed instead of trust. Agents should accelerate workflow creation while analysts retain control of decisions that affect the business. Production governance can't become collateral damage in the race to move faster.
Why pipeline development still takes weeks
The delay usually begins with a familiar handoff. An analyst submits a request. An engineer interprets the requirements, builds the pipeline, and hands the output back. The analyst discovers that a join uses the wrong grain, a filter excludes a valid business case, or the final structure doesn't support the intended report. The request returns to engineering, and the cycle starts again.
No one in that loop is failing. The analyst understands the business need, and the engineer understands the platform. The tooling forces them to pass context through tickets, meetings, and code reviews. Each handoff creates room for delay and misinterpretation.
The backlog compounds the problem. Every new data source, report, and business question adds another request, while engineering capacity remains finite. Routine changes compete with platform maintenance, complex projects, and production incidents.
Validation also happens too late. If analysts can inspect pipeline logic only by reading code, errors may not become visible until the data reaches a dashboard or financial model. The problem isn't that analysts lack skill. Many know SQL, and all bring domain knowledge the engineer doesn't have. The problem is separating the person defining the logic from the person building and validating it.
What agentic pipeline development actually means
Agentic data preparation uses AI agents to create first-draft visual data workflows from natural-language descriptions. The analyst reviews, tests, and refines the result before anything reaches production. The agent handles syntax, scaffolding, and repetitive assembly. The analyst owns business logic and validation.
Much of the agentic-pipeline conversation concerns operations: monitoring, self-healing orchestration, and failure recovery, or what happens after a pipeline runs. We're focused on creation: turning business intent into an inspectable, production-ready workflow.
The work still follows three universal steps:
- Find the right data
- Understand what that data represents and how it should be used
- Transform it for analytics or AI
Agents can read available schemas and metadata, assemble an initial workflow, and respond to requested changes. But they don't know why finance treats a null as zero in one report and as missing data in another, or which customer definition executives approved last quarter.
Neither the agent nor the analyst should work alone. The agent supplies speed. The analyst supplies context, judgment, and accountability.
Why full automation breaks down in data preparation
We don't believe agents should generate an entire data workflow and automatically deploy it to production without review. Data preparation contains too many domain decisions disguised as technical steps.
Consider a join. The syntax is mechanical, but choosing the correct tables, keys, and grain requires understanding what each row means. A misconfigured join might run successfully while silently dropping half the records. Nothing crashes. The error appears later as a plausible but incorrect metric.
Data types create the same risk. An agent might infer a type that looks reasonable from sample records but skews an entire financial model when applied to the full dataset. Null handling, duplicate treatment, effective dates, and exception rules also depend on business context that doesn't live neatly in a schema.
These are domain decisions, not syntax decisions. AI has improved sharply on text-to-SQL, yet the best systems still trail human experts on complex work. On the BIRD benchmark, human data engineers reach 92.96% execution accuracy, while the top-ranked systems in 2026 top out around 82%, and the gap widens on the hardest queries.
Think of building data workflows like assembling a car. Robots play a huge role, but quality gates and human engineers remain at critical checkpoints. Agents should accelerate the mechanical work, while people validate decisions that carry business risk. A visual workflow makes that review practical by letting analysts inspect transformations and compare inputs with outputs before a silent error reaches production.
The generate, refine, deploy workflow
The useful agentic lifecycle isn't prompt and publish. It is Generate, Refine, and Deploy, with the analyst owning the full loop.
Generate: The analyst describes the data preparation goal in natural language. The agent reads the schemas and metadata available under that user's permissions, then creates a first-draft visual workflow. Joins, filters, transformations, and output structure appear on the canvas.
Refine: The analyst reviews the workflow visually. Each step shows what data comes in, which transformation applies, and what comes out. If the logic doesn't match the business definition, the analyst can edit the canvas or describe the change in plain English. Code access, data previews, and restore points remain available for testing and validation.
Deploy: Once validated, the visual workflow compiles to production-grade Spark or SQL and deploys through the organization's governed platform processes, including Git-backed version control, testing, documentation, access controls, and audit trails.
The analyst owns the routine workflow from intent to production. Engineering remains involved for complex architecture, high-risk transformations, platform standards, and exceptional cases. This is governed self-service: analysts get to production inside boundaries set by the data platform team.
How Prophecy puts this into practice
We've built Prophecy around this division of responsibility. Prophecy's AI agents create visual data workflows from natural-language intent, while the analyst remains responsible for validating logic, data correctness, and production readiness.
The refine step works across three connected interfaces:
- Visual canvas: Analysts inspect and edit transformation logic and view the data at each stage
- Code view: SQL-proficient analysts and engineers can inspect or edit the underlying implementation
- Text and chat interface: Users can ask the agent to generate or modify steps throughout development
These are views of the same code-backed workflow. Prophecy's bidirectional interface keeps visual and code representations synchronized.
Prophecy workflows run natively on Databricks, Snowflake, or BigQuery. Transformations execute on the customer's cloud data platform rather than on Prophecy itself; orchestration, ingestion, and egress may use Prophecy Automate, and Prophecy says customer data doesn't leave the customer's security boundary. Prepared data stays in the platform, where analysts and business users can query it directly.
Production readiness can't be an afterthought. Workflows can use Git version control, while built-in security and testing support role-based access control, platform permissions, validation, and governed operation. Prophecy's specialized agents can also generate workflow documentation, keeping production logic and its explanation in sync.
The data platform team sets the guardrails; analysts build within them. Governance means a single source of truth under the platform's access controls, not routing every routine business request through an engineering backlog.
The meaningful shift isn't from manual work to full automation. It's from engineering-dependent development to analyst-led development with production-grade governance intact. When analysts can generate, validate, and deploy their own data workflows inside boundaries controlled by the data platform team, engineers can focus on work that truly requires engineering while analysts respond to business questions without sacrificing trust.
Book a demo to see how Prophecy supports analyst-led data workflow development from first draft through governed production.
Ready to see Prophecy in action?
Request a demo and we’ll walk you through how Prophecy’s AI-powered visual data pipelines and high-quality open source code empowers everyone to speed data transformation
