P

Prodapt

Telecom-focused consulting and managed services

Senior Databricks Solution Architect

Data EngineeringConfirmed live in the last 24 hours
Full-timeDallas, TX, USANo salary listed

About the Job

Requirements

  • A bachelor's or master's degree in Computer Science, Engineering, Information Technology, or a related discipline.
  • 14 or more years of experience in data engineering or data architecture.
  • 5 or more years of experience working with modern cloud data platforms.
  • Strong hands-on Databricks experience.
  • Proven experience designing enterprise-scale data platforms.
  • Experience leading technical teams and architecture initiatives.
  • Ability to translate business requirements into scalable technical architectures.

Responsibilities

  • Define the target-state Databricks architecture for the migrated application.
  • Design scalable and secure Lakehouse architecture using Databricks.
  • Define the architecture across Bronze, Silver, and Gold layers.
  • Design data ingestion, transformation, processing, and serving patterns.
  • Define standards for Delta Lake, Unity Catalog, compute, storage, workloads, and environments.
  • Design solutions for batch and near-real-time data processing.
  • Analyze the existing Palantir Foundry application, including its pipelines, datasets, applications, workflows, and AI capabilities.
  • Create a feature-by-feature mapping between existing Palantir capabilities and Databricks capabilities.
  • Identify capabilities that can be migrated directly versus those requiring redesign.
  • Define migration patterns for data ingestion pipelines, data transformation pipelines, data quality and validation, data comparison and reconciliation, AI/LLM-based information extraction, dashboards and reporting, chatbot capabilities, business workflows and actions, and document processing.
  • Develop migration roadmaps, architecture decisions, and technical dependencies.
  • Identify opportunities to improve scalability, maintainability, and performance during migration.
  • Design robust data pipelines using Lakeflow / Spark Declarative Pipelines, Apache Spark, and Delta Lake.
  • Design ingestion from databases, APIs, files, documents, and other enterprise data sources.
  • Establish data quality, validation, lineage, and observability mechanisms.
  • Design incremental processing and change-data-capture patterns where required.
  • Optimize Delta tables, Spark workloads, and Databricks compute for performance and cost.
  • Architect solutions for extracting information from structured and unstructured documents.
  • Design retrieval-augmented generation (RAG) and large language model (LLM)-based solutions where applicable.
  • Evaluate Databricks capabilities for document processing, information extraction, embeddings and vector search, LLM inference, model serving, and AI agents or conversational applications.
  • Define approaches for migrating existing Palantir AIP-based extraction capabilities into Databricks.
  • Ensure AI solutions are scalable, secure, explainable, and production-ready.
  • Design the target architecture for dashboards, drill-down reporting, and analytical applications.
  • Define semantic/data serving layers required by business applications.
  • Architect chatbot and conversational analytics solutions.
  • Design integration between Databricks and downstream applications/APIs.
  • Implement enterprise data governance using Unity Catalog.
  • Define catalog, schema, table, volume, and access-control strategies.
  • Design role-based access control (RBAC) and data-access policies.
  • Define data lineage, auditing, and governance standards.
  • Ensure compliance with enterprise security and privacy requirements.
  • Lead a team of data engineers, ML/AI engineers, and developers.
  • Conduct architecture reviews and technical design sessions.
  • Create high-level and low-level design documents, architecture diagrams, and migration plans.
  • Establish Databricks development standards and best practices.
  • Review code and implementation approaches.
  • Work closely with business stakeholders, Palantir subject-matter experts, cloud architects, and application teams.
  • Troubleshoot complex architectural and performance issues.

About the Company

Prodapt

Telecom-focused consulting and managed services

Prodapt focuses on the telecommunications, media, and technology sector (the 'Connectedness' industry) by providing specialized consulting and managed services. It helps clients through a service-based model that includes strategic consulting, application development, systems integration, and ongoing operational support. Its offerings cover digital transformation, cloud and network services, fiber rollout, 5G enablement, cloud migration, and applying AI to network operations to modernize telecom operators and digital platform companies. What sets Prodapt apart is its deep, industry-specific focus on the Connectedness vertical, allowing it to build tailored frameworks and solutions rather than generic IT services, backed by global reach across the Americas, Europe, India, and Africa. The company aims to be a strategic partner that helps clients navigate the evolving digital landscape, accelerate deployment of new services, and optimize network operations.

Company size
5,001-10,000
Company stage
Growth Equity (Venture Capital)
Total funding
$75M
Headquarters
Tamil Nadu, India
Founded
1999

Recently Posted Jobs

Explore more opportunities at Prodapt

See more jobs