Full-Time

Staff AI/ML Engineer

Prophecy

Prophecy

51-200 employees

Low-code data engineering for Spark/Airflow

Compensation Overview

$250k - $350k/yr

+ Equity

San Francisco, CA, USA

Hybrid

Hybrid role with 2-3 days in office per week; varies week to week.

Category
AI & Machine Learning (1)
Required Skills
LLM
Kubernetes
Python
Machine Learning
Java
ETL
RAG
Salesforce
Microservices
AWS
Go
Scala
REST APIs
LangChain
Data Analysis
Requirements
  • Hands-on LLM/agent building (e.g., LangChain/Graph, CrewAI) and tuning to quality benchmarks
  • Experience with semantic search, Retrieval-Augmented Generation, and vector databases
  • Experience with prompt engineering and optimization
  • Agentic Use Cases: Workflow Orchestration: Automating multi-step processes (e.g., ETL pipelines, supply chain optimization, or CRM workflows like Salesforce Agentforce)
  • Agentic Use Cases: Code Generation/Assistance: Developing or debugging code autonomously (e.g., similar to Devin AI or GitHub Copilot extensions)
  • Agentic Use Cases: Data Analysis/Transformation: Querying, manipulating, or analyzing data across sources (e.g., topic extraction, document analytics in vector DBs like Weaviate)
  • Agentic Use Cases: Personalized Assistance: Tailoring recommendations or plans (e.g., e-commerce, education, or finance agents)
  • Agentic Use Cases: Autonomous Decision-Making: Real-time decisions in domains like trading or healthcare (e.g., crypto trading, medical workflow optimization)
  • Agentic Use Cases: Content Creation: Generating text, art, or media (e.g., social media automation, presentation creation)
  • Agentic Use Cases: Multi-Agent Systems: Collaborative agents for simulations or transactions (e.g., policy-constrained evaluation, machine-to-machine commerce)
  • Versatile software developer: fluency in Python, REST APIs, microservices in public cloud; some experience with Go, AWS, k8s, java, Scala
  • Builder mentality: Demonstrated experience taking ideas to production
  • Experience: 6+ years of Software Engineering experience minimum with 2+ of these being in AI/ML
  • Location: Hybrid role with 2-3 days a week in office. However, this can vary week to week
  • Seniority: Staff or Principal
Desired Qualifications
  • Expertise in building and managing Knowledge Graphs, including entity extraction, linkage, and ontology design
  • Experience integrating Knowledge Graph with AI systems for retrieval, reasoning, and agent context (e.g., enriching with LLM)
  • ML/LLM work in code generation (e.g., Codex, text-to-SQL), semantic extraction, or knowledge graphs (e.g., Neo4j, Neptune)
  • Experience with big-data engines like Spark
  • Compiler development for languages like SQL, Python, or Scala
  • Optimization of ML models for low-latency, high-throughput production use
  • Contributions to open-source AI/ML projects (e.g., Hugging Face, PyTorch)
  • Expertise in retrieval systems or vector databases (e.g., Pinecone, Weaviate)
  • Skill in evaluating tech and driving build/buy decisions

Prophecy.io is a low-code data engineering platform that helps data teams build and manage Spark workflows and Airflow schedules. It offers a visual interface for designing jobs and schedules, plus metadata search and column-level lineage for governance. It targets medium to large enterprises and uses a subscription-based model for access to its toolkit. Its goal is to speed up development and improve reliability and observability of data pipelines.

Company Size

51-200

Company Stage

Series B

Total Funding

$116.5M

Headquarters

San Francisco, California

Founded

2017

Your Connections

People at Prophecy who can refer or advise you

Simplify Jobs

Simplify's Take

What believers are saying

  • Powers tens of thousands of pipelines for Fortune 500 companies across finance, healthcare, retail, technology.
  • Series B1 funding of $47M from Smith Point Capital, HSBC, JPMorgan Chase accelerates product expansion.
  • Claude Code agents democratize data prep for non-technical analysts, expanding addressable market beyond engineers.

What critics are saying

  • Databricks bundles native AI data prep tools, eroding Prophecy's value as standalone subscription add-on.
  • Snowflake Cortex Analyst generates SQL workflows natively, reducing dependency on external low-code platforms.
  • Anthropic enables Claude Code deployment directly via APIs, rendering Prophecy's proprietary wrapper obsolete.

What makes Prophecy unique

  • AI agents generate visual, inspectable workflows from business intent on Databricks, Snowflake, BigQuery.
  • Two-way visual-code editor keeps Git-stored production code synchronized with user refinements.
  • Unified self-service platform eliminates siloed systems, reducing rework and errors across data teams.

Help us improve and share your feedback! Did you find this helpful?

Benefits

Health Insurance

Unlimited Paid Time Off

Wellness Program

Professional Development Budget

Company Equity

Life Insurance

FSA/HSA

Long Term Disability

Growth & Insights and Company News

Headcount

6 month growth

-1%

1 year growth

1%

2 year growth

1%
PR Newswire
Feb 24th, 2026
Prophecy launches v4 with AI agents for visual data prep on Databricks, Snowflake and BigQuery

Prophecy has launched v4, an AI data preparation and analysis platform that uses AI agents to convert business requirements into visual data workflows. The platform generates inspectable workflows that run natively on Databricks, Snowflake and BigQuery. The system addresses validation challenges in AI-generated data logic by making outputs visual and reviewable, rather than requiring users to check lengthy SQL or Spark code. Workflows are stored as production-grade code in Git and inherit governance policies from underlying data platforms. Prophecy v4 aims to replace legacy desktop tools by combining AI-driven productivity with cloud-native execution. The platform unifies data analysis steps into a single interface where users can iteratively refine results through conversational interactions with agents whilst maintaining synchronisation between visual workflows and code. The platform is available now with free accounts offered.

Taber Communications
Mar 28th, 2025
Prophecy 4.0 Offers Fully Governed Self-Service Data Prep for Databricks SQL

Additionally, Prophecy has introduced built-in automation with a drag-and-drop interface.

Prophecy
Jan 20th, 2025
Prophecy takes in $47M to scale-up and re-imagine data integration with AI

I’m pleased to share that Prophecy today announced a $47M Series B1 round. Smith Point Capital led the round, with HSBC joining as a new investor and participation from existing investors including Berkeley SkyDeck, DallasVC, Insight Partners, JPMorgan Chase and SignalFire.

FinSMEs
Jan 16th, 2025
Prophecy Raises $47M in Series B Extension Funding

Prophecy raises $47M in Series B extension funding.

SiliconANGLE Media
Jan 16th, 2025
Prophecy raises $47M for AI data pipelines

Prophecy Inc., a data copilot startup, raised a $47M Series B extension led by Smith Point Capital, with participation from HSBC and others. The company uses generative AI to automate data pipeline development, making data accessible across systems. Prophecy's copilot, integrated with Databricks, accelerates AI initiatives by simplifying data preparation. The funding will enhance its platform and expand its customer base by 2025.