Agent-Powered Pipeline Development: Designed for Humans, Built for Production

AI agents are having a moment. From writing code to booking travel, they promise to take on the grunt work so we can move faster and focus on what matters.

But when it comes to building production-ready data pipelines, the story isn't so simple.

AI agents are transforming work. But they aren’t one-size-fits-all

The promise of agents is real. They boost productivity, act like intelligent helper functions, and take on repetitive, low-level tasks so humans don’t have to. They’re the factory robots of digital work. Relentless. Precise. Fast.

In some industries, they're already making a huge difference:

• In customer support, agents can instantly draft email replies, suggest responses, or triage tickets based on priority.
• In software development, coding agents can scaffold entire applications from natural language prompts.

But the success of agents always depends on the domain and the design of human-AI collaboration.

Production pipelines are high-stakes. You can’t just hit "Go."

We don’t believe agents should generate an entire pipeline and automatically deploy it to production.

Why?

Because data prep isn’t a one-shot task. The stakes are high. And the risks are very real:

• An incorrect join might silently drop half your rows
• An assumption about column types could throw off your entire model
• Missed edge cases can lead to wrong metrics. Or worse, wrong business decisions.

Data prep involves judgment. Nuance. Context. And too many interdependent steps to trust an opaque black box.

The better path: Step-by-step assistance, with humans in the loop

Think of building pipelines like assembling a car. Robots play a huge role. But there are checkpoints, quality gates, and human engineers reviewing critical steps.

That’s how we’re building agents at Prophecy: To simplify, accelerate, and assist. But always with a human at the center, ready to inspect and refine.

This approach empowers more users, scales trust, and helps organizations balance even faster self-service with governance.

From first principles: How we designed our agents

To design AI agents for data preparation, we started from the ground up:

What does it actually take to create a production-ready pipeline? We broke the problem down into three universal steps:

1. Find the right data
2. Understand what that data is and how to use it
3. Transform it to make it AI- and analytics-ready

This is true whether you're a data engineer or a business analyst. Whether you're on the finance team or customer service. For BI or AI use cases. With desktop files or enterprise databases. These steps never change. 

What does change is how efficiently (and confidently) you can move through them. That’s where agents come in.

See it. Adjust it. Talk to it. The power of visual + natural language

Prophecy’s unique approach keeps every step transparent, visible, and editable. As the agent works, you see its output rendered visually on the canvas. You can click into each transformation, inspect its logic, and view the data at that stage. 

And if something needs changing? Just ask. You can refine any step using natural language - not just at the start, but throughout the process. Want to rename a column? Filter rows differently? Update a join condition? 

Say it in plain English, and the agent will respond. No black boxes.

This makes data prep accessible to more users, while keeping quality and governance intact.

How Prophecy agents help you build pipelines: Step-by-step

Let’s walk through how Prophecy agents support each step of the pipeline journey:

Find the right data

Locating the right dataset is often the most time-consuming and overlooked step in building data pipelines. Users struggle to:

• Search across diverse sources: Siloed datasets and inconsistent metadata make it hard to know what’s available.
  
  
• Semantic search gaps: Traditional keyword search is limited. Users need to find data based on meaning, not just names or descriptions.

Prophecy provides search capabilities across all accessible datasets in your fabric - leveraging semantic matching and enabling you to search by title, description, or even column names. This drastically reduces time spent hunting for the right data, so your team can focus on higher value activities.

Guardrails are still in place—the agents are limited by the permissions of the users. With our Databricks Unity Catalog integration, you can extend users access to the power of agents, without creating security or governance risks. 

Cut the time it takes to find the right data.

‍

Understand the data

Even after users have found candidate datasets, the next hurdle is exploring and understanding what each offers:

• Limited dataset insight: Users rarely have the full context—schema details, sample records, or rich descriptions—needed to assess whether a dataset is fit for purpose.
  
  
• Comparing alternatives: Choosing between datasets is difficult without side-by-side comparisons of details completeness and relevance.

With Prophecy, users can preview data, understand schema details, read descriptions, and even ask questions about any dataset to quickly evaluate its suitability. Comparing datasets helps users select the most relevant data sources for their purpose. 

With this, exploration to understand the data becomes a natural & transparent process—empowering all users to quickly make informed selections.

Choose the right dataset faster with side-by-side comparisons.

‍

Make more informed selections by quickly exploring datasets.

‍

Transform it for analytics or AI

Once the right data is in hand, users often hit roadblocks in shaping it to fit their unique needs:

• Transformation logic: Users may know what they want but struggle to translate business logic into data transformation steps.
  
  
• Feedback and control: It’s hard to see the impact of each transformation or easily adjust logic as requirements evolve.

Simply describe your transformation goals in natural language and Prophecy translates them into visual, editable pipeline steps. The agent is designed for maximum transparency. See real-time visual feedback and easily tweak steps—either by updating instructions or making direct changes in the pipeline canvas. You’re always in control, with guardrails on what questions the user can ask the agent and the flexibility to refine and iterate until the output is just right.

From natural language to visual pipelines. Refine each transformation easily and with full transparency.

‍

Inspect: See how each transformation step changes data, with clear expressions that produced the change as well as before-and-after views of the data. This helps in building confidence that the right outcome was achieved and in case there are errors, it becomes easier to identify and fix the issues.

Fix issues quickly and trust the outcome with full visibility before and after every step.

‍

Preview/restore: Exploring data and refining transformations is an iterative process. Users often try out different prompts and actions to achieve the desired results, and flexibility is essential. That’s why our solution includes Preview and Restore options:

• Preview: Instantly see the effects of your changes—whether you’re experimenting with new prompts or adjusting transformation logic. This gives you confidence in every step before committing to changes. In the same vein, the ability to quickly preview an older state allows users to understand & then decide if they want to go to that state.
  
  
• Restore: If you want to revisit or revert to a previous version of your work, simply restore your session to any earlier state in your chat history. This makes it easy to backtrack, try alternative approaches, or restart from a known good configuration.

Rewind and refine with ease. Jump back to any point in your workflow to explore or correct.

‍

Data preparation is often a means to the end of performing analytics and tracking the analysis on dashboards. With Prophecy, users can create rich visualizations for their final output & publish to business apps, thus completing the task.

Turn prepared data into insights by creating visualizations and publishing to business apps - all in one flow.

‍

The future is assistive

When it comes to building production-ready pipelines, agents are most powerful not as replacements for people, but as tools that empower them to work faster and with greater confidence.

That’s why we’ve designed agents that:

• Work in real time, transparently showing their steps on a visual canvas
• Invite feedback and updates through natural language
• Operate under your governance, your standards, and your approval process

This is how we deliver on the promise of governed, self-service data preparation. Faster pipelines. Greater accuracy. And a data platform that works the way you think.