Want to know if this job is worth applying to?
Bengaluru, Karnataka, India
Hybrid
The leader in AI-native data preparation and analysis, Prophecy is revolutionizing how the world's top enterprises turn data chaos into reliable insights. We introduce the AI-native data lifecycle (generate, refine, deploy) where our industry leading AI agents and humans work hand-in-hand in visual and document interfaces to analyze, transform and prepare data, to ship trusted insights at enterprise scale. To learn more, visit us on LinkedIn.
This is an exceptional opportunity to be a technical leader on the team that powers the heart of Prophecy: the Runtime, our orchestration and execution service. Every data workflow , whether generated by our AI data agents or built by a human in the visual editor , is planned, scheduled, and run at enterprise scale by Runtime, directly against the customer's own platform (Databricks, Snowflake, BigQuery, and more). When a business user asks an agent a question and sees trusted results seconds later, Runtime is what made it happen.
As a Staff Engineer, you will set the technical direction for this service: the distributed execution engine, the task coordinator, the multi-engine connectors, and the fast interactive previews that make the AI-native experience feel instant. You will own hard, ambiguous, high-leverage problems end-to-end , from architecture to production reliability , and raise the bar for the engineers around you.
We're seeking A players to work with A players. We're a high-growth company on a once-in-a-lifetime journey to revolutionize the way data is prepared and analyzed. Be a part of it.
Own the engine behind the product. Architect and evolve the runtime service that turns agent- and human-authored data workflows into high-performance jobs that run reliably in production and interactively during authoring. This is deep-tech at the center of the platform.
Make it fast and make it scale. Design the distributed execution and task-coordination layer , scheduling, concurrency, resource management, streaming of sample data (interims) , so that pipelines run efficiently from a single interactive preview to large production workloads.
Run everywhere the customer's data lives. Extend execution across multiple engines and warehouses , Databricks, Snowflake, BigQuery, Spark, dbt, Airflow , with first-class connectors, query pushdown, and consistent behavior across clouds.
Power the AI data agents. Build the execution path that lets our agents generate, run, validate, and refine pipelines in a tight loop , the difference between a plausible answer and a trusted one.
Own reliability and observability. Drive production robustness: fault tolerance, graceful degradation, recovery and restartability, monitoring, and the SLAs that enterprise customers depend on.
Lead technically. Set architecture and standards, drive cross-team designs, and mentor engineers. Own features and initiatives end-to-end, as a founder would , from the requirement to delivery. We don't believe in hierarchy or years of experience for their own sake; we believe in ownership and impact.
What You’ll Need
Preferred Qualifications
What You’ll Have At Prophecy
