Full-Time

Staff Machine Learning Engineer

Knowledge Graph

Posted on 9/18/2025

Prophecy

Prophecy

51-200 employees

Low-code data engineering for Spark/Airflow

Compensation Overview

$250k - $350k/yr

San Francisco, CA, USA

Hybrid

Category
AI & Machine Learning (2)
,
Required Skills
LLM
Kubernetes
Python
Java
Microservices
AWS
Go
Scala
REST APIs
Requirements
  • Expertise in building and managing Knowledge Graphs, including entity extraction, linkage, and ontology design.
  • Experience integrating Knowledge Graph with AI systems for retrieval, reasoning, and agent context (e.g., enriching with LLM).
  • Tech lead experience: Owning end-to-end KG projects, from design to deployment, with leadership in cross-functional teams.
  • Versatile software developer: fluency in Python, REST APIs, microservices in public cloud; some experience with Go, AWS, k8s, java, Scala.
  • Hands-on experience in developing, optimizing and operating neo4j or similar graph databases.
  • Builder mentality: Demonstrated experience taking ideas to production.
  • Seniority: Senior to Staff Level.
  • Experience: 5+ years in industry.
Responsibilities
  • Own end-to-end knowledge-graph projects from design to deployment, with leadership in cross-functional teams.
  • Build and manage knowledge graphs including entity extraction, linkage, and ontology design, and integrate with AI systems for retrieval, reasoning, and agent context.
  • Develop, optimize, and operate graph databases such as Neo4j or similar in production environments.
  • Collaborate with cross-functional teams to deliver end-to-end data preparation and analysis platform features.
  • Demonstrate a builder mentality by taking ideas from concept to production.
Desired Qualifications
  • ML/LLM work in code generation (e.g., Codex, text-to-SQL), semantic extraction, or knowledge graphs (e.g., Neo4j, Neptune).
  • Experience with big-data engines like Spark.
  • Compiler development for languages like SQL, Python, or Scala.
  • Optimization of ML models for low-latency, high-throughput production use.
  • Contributions to open-source AI/ML projects (e.g., Hugging Face, PyTorch).
  • Expertise in retrieval systems or vector databases (e.g., Pinecone, Weaviate).
  • Skill in evaluating tech and driving build/buy decisions.

Prophecy.io is a low-code data engineering platform that helps data teams build and manage Spark workflows and Airflow schedules. It offers a visual interface for designing jobs and schedules, plus metadata search and column-level lineage for governance. It targets medium to large enterprises and uses a subscription-based model for access to its toolkit. Its goal is to speed up development and improve reliability and observability of data pipelines.

Company Size

51-200

Company Stage

Series B

Total Funding

$116.5M

Headquarters

San Francisco, California

Founded

2017

Simplify Jobs

Simplify's Take

What believers are saying

  • Powers tens of thousands of pipelines for Fortune 500 companies across finance, healthcare, retail, technology.
  • Series B1 funding of $47M from Smith Point Capital, HSBC, JPMorgan Chase accelerates product expansion.
  • Claude Code agents democratize data prep for non-technical analysts, expanding addressable market beyond engineers.

What critics are saying

  • Databricks bundles native AI data prep tools, eroding Prophecy's value as standalone subscription add-on.
  • Snowflake Cortex Analyst generates SQL workflows natively, reducing dependency on external low-code platforms.
  • Anthropic enables Claude Code deployment directly via APIs, rendering Prophecy's proprietary wrapper obsolete.

What makes Prophecy unique

  • AI agents generate visual, inspectable workflows from business intent on Databricks, Snowflake, BigQuery.
  • Two-way visual-code editor keeps Git-stored production code synchronized with user refinements.
  • Unified self-service platform eliminates siloed systems, reducing rework and errors across data teams.

Help us improve and share your feedback! Did you find this helpful?

Benefits

Health Insurance

Unlimited Paid Time Off

Wellness Program

Professional Development Budget

Company Equity

Life Insurance

FSA/HSA

Long Term Disability

Growth & Insights and Company News

Headcount

6 month growth

0%

1 year growth

1%

2 year growth

3%
PR Newswire
Feb 24th, 2026
Prophecy launches v4 with AI agents for visual data prep on Databricks, Snowflake and BigQuery

Prophecy has launched v4, an AI data preparation and analysis platform that uses AI agents to convert business requirements into visual data workflows. The platform generates inspectable workflows that run natively on Databricks, Snowflake and BigQuery. The system addresses validation challenges in AI-generated data logic by making outputs visual and reviewable, rather than requiring users to check lengthy SQL or Spark code. Workflows are stored as production-grade code in Git and inherit governance policies from underlying data platforms. Prophecy v4 aims to replace legacy desktop tools by combining AI-driven productivity with cloud-native execution. The platform unifies data analysis steps into a single interface where users can iteratively refine results through conversational interactions with agents whilst maintaining synchronisation between visual workflows and code. The platform is available now with free accounts offered.

Taber Communications
Mar 28th, 2025
Prophecy 4.0 Offers Fully Governed Self-Service Data Prep for Databricks SQL

Additionally, Prophecy has introduced built-in automation with a drag-and-drop interface.

Prophecy
Jan 20th, 2025
Prophecy takes in $47M to scale-up and re-imagine data integration with AI

I’m pleased to share that Prophecy today announced a $47M Series B1 round. Smith Point Capital led the round, with HSBC joining as a new investor and participation from existing investors including Berkeley SkyDeck, DallasVC, Insight Partners, JPMorgan Chase and SignalFire.

FinSMEs
Jan 16th, 2025
Prophecy Raises $47M in Series B Extension Funding

Prophecy raises $47M in Series B extension funding.

SiliconANGLE Media
Jan 16th, 2025
Prophecy raises $47M for AI data pipelines

Prophecy Inc., a data copilot startup, raised a $47M Series B extension led by Smith Point Capital, with participation from HSBC and others. The company uses generative AI to automate data pipeline development, making data accessible across systems. Prophecy's copilot, integrated with Databricks, accelerates AI initiatives by simplifying data preparation. The funding will enhance its platform and expand its customer base by 2025.

INACTIVE