AI/ML Engineer

The Opportunity

We're building the AI engine behind root cause analysis for a platform that supports over 100 enterprise customers — this role owns everything from the vector database to the confidence scoring that decides whether an engineer gets notified automatically or a human steps in.

What You'll Contribute

• Design and operate the pgvector schema across all collections (past incidents per tenant, runbooks, operational context, CI/CD change events), owning indexing strategy, query performance tuning, and strict data isolation across 100+ tenants with zero cross-tenant leakage.

• Build and maintain embedding pipelines that convert operational documents into searchable vector representations, selecting embedding models and designing chunking strategies for structured JSON, unstructured prose, and time-series data types.

• Build the RAG retrieval layer that runs at incident time, tuning relevance thresholds, re-ranking strategies, and hybrid (dense and sparse) search to surface the most contextually relevant documents per collection.

• Design the full ingestion lifecycle, including runbook imports, historical incident backfill, CI/CD post-deploy event hooks, and the feedback loop that embeds resolved incidents back into the vector database.

• Design and iterate the prompt that assembles Grafana's stateless RCA, live signal data, and RAG-retrieved context into a single input, instructing Claude to reason across evidence types and produce a structured, actionable RCA readable in under 30 seconds.

• Define and maintain the JSON schema for RCA output (root cause, confidence score, contributing factors, immediate actions, SLA impact, precedent reference, escalation path) and build the Pydantic validation layer that catches schema violations before delivery.

• Own the calibrated 0–1 confidence scoring framework weighting RAG precedent strength, signal consistency, deployment correlation, runbook recognition, root cause specificity, and customer context fit, and build the FastAPI gate logic that routes RCAs to the correct action tier.

• Maintain a golden test set of 20–30 historical incidents and run the full pipeline against it after every prompt or RAG change, using a judge-prompt approach to score accuracy and catch regressions.

• Run monthly calibration checks comparing confidence scores against SRE-confirmed outcomes, calculating Expected Calibration Error (ECE) and adjusting thresholds accordingly.

• Build a prompt versioning system with changelog tracking, and monitor token usage, cost per RCA, and API latency, alerting on unexpected cost trends.

• Track retrieval quality metrics including relevance scores, coverage gaps, cross-tenant isolation verification, and ablation testing of RCA quality with and without each data collection.

What We're Seeking

• 4+ years of experience in AI/ML engineering, data engineering, NLP, or a closely related field, including hands-on LLM work.

• Production experience building and operating a vector database (pgvector, Pinecone, Weaviate, or equivalent).

• Strong understanding of embedding models and how embedding quality affects retrieval accuracy.

• Experience designing and operating data ingestion pipelines at scale (Celery, Airflow, or similar).

• Demonstrated experience prompt engineering for production LLM systems requiring reliable structured outputs, not just chatbots or demos.

• Experience building LLM evaluation frameworks, including golden test sets, judge prompts, and accuracy metrics.

• Strong Python skills, including FastAPI, Pydantic, SQLAlchemy, and async patterns.

• Clear understanding of how RAG context affects LLM reasoning and how to diagnose retrieval-driven errors.

• Familiarity with semantic search concepts: cosine similarity, approximate nearest neighbor, and hybrid search.

• Production experience with the Claude API (messages format, system prompts, structured output patterns) is a plus.

• Experience with LLM output validation frameworks such as Pydantic or Guardrails is a plus.

• Understanding of confidence calibration techniques (ECE, Platt scaling, temperature effects) is a plus.

• Exposure to SRE or observability domains, and familiarity with multi-tenant architectures where tenant isolation is a hard requirement, is a plus.

• Knowledge of re-ranking models (cross-encoders) and experience with Kubernetes/AWS EKS is a plus.

Our Offer to You

• An inclusive culture strongly reflecting our core values: Act Like an Owner, Delight Our Customers and Earn the Respect of Others.

• The opportunity to make an impact and develop professionally by leveraging your unique strengths and participating in valuable learning experiences.

• Highly competitive compensation, benefits and rewards programs that encourage you to bring your best every day and be recognized for doing so.

• An engaging, people-first work environment offering work/life balance, employee resource groups, and social events to promote interaction and camaraderie.