Senior AI Engineer - Claude & Databricks

Senior AI Engineer – Claude & Databricks

Position Summary

We are seeking a Senior AI Engineer with deep experience building production-grade AI solutions using Claude, combined with a strong background in data engineering and modern data platforms, including Databricks.

This role will design, build, and operationalize enterprise AI applications that integrate large language models with governed enterprise data. The ideal candidate combines AI/LLM engineering, software development, and data engineering, and is comfortable taking solutions from architecture and experimentation through production deployment.

This is a hands-on senior technical role for someone who understands not only how to use an LLM, but how to build reliable, secure, scalable AI solutions around it.

Key Responsibilities

AI & Claude Engineering

• Design and develop enterprise AI applications using Anthropic Claude and Claude APIs.
• Build LLM-powered applications including AI agents, copilots, conversational applications, RAG solutions, document intelligence, and workflow automation.
• Develop effective prompting, structured-output, tool-use, and agentic patterns using Claude.
• Build integrations that allow Claude-based applications to securely interact with enterprise data, APIs, applications, and business processes.
• Implement retrieval-augmented generation (RAG) architectures using structured and unstructured enterprise data.
• Develop evaluation frameworks to measure AI solution quality, accuracy, reliability, and business effectiveness.
• Implement AI guardrails, observability, exception handling, security, and responsible AI practices.
• Optimize AI solutions for performance, reliability, scalability, and cost.
• Evaluate emerging AI patterns and determine where they provide practical business value versus unnecessary complexity.

Data Engineering & Databricks

• Design and develop data pipelines supporting AI, analytics, and machine learning workloads.
• Build scalable data solutions using Databricks, Spark, PySpark, Python, SQL, and Delta Lake.
• Prepare structured and unstructured enterprise data for use by LLM and AI applications.
• Design ingestion, transformation, enrichment, chunking, metadata, and retrieval processes supporting RAG and agentic AI solutions.
• Work with Unity Catalog and enterprise governance frameworks to ensure AI solutions use governed, secure, and traceable data.
• Integrate AI applications with lakehouse, data warehouse, API, document, and operational data sources.
• Apply data quality, lineage, metadata, access control, and data governance standards to AI solutions.

Architecture & Production Engineering

• Define technical architectures for enterprise AI solutions from data ingestion through model interaction and application delivery.
• Make architecture decisions across LLMs, data platforms, retrieval technologies, APIs, orchestration frameworks, and cloud services.
• Design reusable AI services, frameworks, APIs, and components that can support multiple business use cases.
• Establish development, testing, evaluation, deployment, and monitoring patterns for production AI applications.
• Participate in architecture and code reviews and mentor other engineers in AI and data engineering practices.
• Work directly with business, data, architecture, security, governance, and engineering teams to translate business problems into implementable AI solutions.

Required Qualifications

• 8+ years of software, data engineering, analytics engineering, or related technology experience, with significant recent experience building AI/ML or GenAI solutions.
• Demonstrated, hands-on experience developing applications using Anthropic Claude.
• Strong experience integrating commercial LLMs through APIs into production applications.
• Experience developing RAG, agentic AI, tool-calling, or LLM-based enterprise applications.
• Strong programming experience with Python.
• Significant data engineering experience, including:
  • Databricks
  • Apache Spark / PySpark
  • SQL
  • Delta Lake
  • ETL / ELT
  • Data pipelines and integration
• Experience designing solutions that combine LLMs with enterprise data.
• Understanding of embeddings, vector search, semantic retrieval, context management, and document processing.
• Experience with API development and application/system integration.
• Strong understanding of data governance, data quality, metadata, lineage, security, and access controls.
• Experience with cloud platforms such as Azure, AWS, or Google Cloud.
• Experience implementing CI/CD and modern software engineering practices.
• Ability to independently take a complex AI use case from concept and architecture through working production implementation.

Preferred Qualifications

• Experience with Claude Code in software engineering and development workflows.
• Experience with Model Context Protocol (MCP) and tool-enabled/agentic architectures.
• Experience with Databricks Unity Catalog and governed enterprise data environments.
• Experience building AI applications against large enterprise data estates.
• Experience with vector databases/vector search, embeddings, document processing, and semantic search.
• Experience modernizing traditional analytics or data platforms to support AI use cases.
• Familiarity with MLflow or similar model/application lifecycle and evaluation capabilities.
• Experience with Azure DevOps, GitHub, or comparable CI/CD platforms.
• Experience implementing AI governance, security, evaluation, and monitoring standards.

What We're Looking For

We are looking for someone who sits at the intersection of three disciplines:

AI Engineer
Deeply understands Claude, LLM application patterns, RAG, agents, tool use, evaluation, prompt/context engineering, and production AI.

Data Engineer
Understands how enterprise data is acquired, transformed, governed, secured, and made usable for AI, with meaningful hands-on Databricks experience.

Solution Architect / Senior Engineer
Can take an ambiguous business problem, define an appropriate AI architecture, make pragmatic technology decisions, and personally build the most critical components.

The right candidate should be able to design an AI solution around Claude, understand how the underlying enterprise data needs to be structured and governed in Databricks, and then be hands-on enough to build the solution.