Allata

Strategy, architecture, and enterprise software development

Databricks Manager - Databricks

Data EngineeringConfirmed live in the last 24 hours

Full-timeDallas, TX, USANo salary listedRemote

About the Job

Requirements

Previous experience as an architect or lead technical professional on enterprise data platforms.
Hands-on experience with Databricks technologies, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Structured Streaming.
Strong expertise in Spark (PySpark and/or Scala), Spark SQL, and distributed job optimization.
Solid background in data warehouse and lakehouse design, with practical familiarity with Medallion, Lambda, and Kappa patterns.
Experience integrating SaaS/ETL/connectors such as Fivetran, orchestration platforms such as Airflow, Azure Data Factory, or Data Fabric, and ELT/ETL tooling.
Experience with relational and hybrid databases such as Microsoft SQL Server, PostgreSQL, Oracle, Azure SQL, and AWS RDS/Aurora or equivalents.
Proficiency in CI/CD for data pipelines using Azure DevOps, GitHub Actions, Jenkins, or similar, and packaging/deployment of artifacts such as .whl files and containers.
Experience with batch and streaming processing, file compaction, partitioning strategies, and storage tuning.
Good understanding of cloud security, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking.
Familiarity with observability and monitoring tools such as Prometheus, Grafana, Datadog, native cloud monitoring, or equivalent.

Responsibilities

Define overall data platform architecture for Lakehouse/EDW, including reference patterns, technology selection, and integration blueprint.
Design conceptual, logical, and physical data models for multi-tenant and vertical-specific data products; standardize logical layers for ingest/raw, staged/curated, and serving.
Establish data governance, metadata, cataloging, lineage, data contracts, and classification practices for analytics and machine learning use cases.
Define security and compliance controls, including access management, data masking, encryption in transit and at rest, network segmentation, and audit policies.
Architect scalability, high availability, disaster recovery, and capacity and cost management strategies for cloud and hybrid deployments.
Lead selection and integration of platform components, including Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory/Data Fabric, orchestration, and monitoring/observability.
Design and enforce CI/CD patterns for data artifacts, including notebooks, packages, and infrastructure as code, with testing, automated deployments, and rollback strategies.
Define batch and streaming ingestion patterns, file compaction strategies, partitioning schemes, and storage layouts to optimize input/output and costs.
Specify observability practices, including metrics, service-level agreements, health dashboards, structured logging, tracing, and alerting for pipelines and jobs.
Act as a technical authority and mentor for Data Engineering teams, and perform architecture and code reviews for critical components.
Collaborate with stakeholders including Data Product Owners, Security, Infrastructure, BI, and ML to translate business requirements into technical solutions and roadmaps.
Design, develop, test, and deploy processing modules using Spark (PySpark/Scala), Spark SQL, and database stored procedures where applicable.
Build and optimize data pipelines on Databricks and complementary engines, including SQL Server, Azure SQL, AWS RDS/Aurora, PostgreSQL, and Oracle.
Implement DevOps practices, including infrastructure as code, CI/CD pipelines for ingestion, transformation, tests, and deployment, automated testing, and version control.
Troubleshoot and resolve complex data quality, performance, and availability issues; recommend and implement continuous improvements.

Desired Qualifications

Automation experience with CI/CD pipelines for deployment and integration workflows, including trunk-based development using Azure DevOps, Jenkins, or Octopus.
Advanced proficiency in PySpark for advanced data processing tasks.
Advanced proficiency in Spark workflow optimization and orchestration using tools such as Asset Bundles or DAG orchestration.
Databricks Certified Data Engineer or Databricks Certified Professional Architect certification, or cloud architect/data certifications for AWS, Azure, or GCP.

Allata is a global consulting and technology services firm founded in 2016, with 350+ employees across the US, India, and Argentina. We partner with some of the world's largest enterprises to move AI from idea to production — building intelligent agents, data foundations, and governance frameworks that scale.

Our teams work at the intersection of strategy, engineering, data, and design, helping clients modernize their technology, unlock data value, and create meaningful digital experiences. At Allata, you'll join an agile, cross-functional team that works closely alongside clients — making a real impact and building lasting partnerships.

We are looking for a Databricks Manager with strong experience in enterprise data platform architecture and governance to lead client-facing data platform implementations. This role blends high-impact architectural responsibilities (reference architectures, security, scalability, cost management, operational model) with technical leadership in designing, building, deploying, and optimizing data pipelines and data products on Lakehouse/EDW platforms (with an emphasis on Databricks). You will own the full lifecycle of data products and standardize patterns, tools, and practices across large-scale, vertical-specific implementations for our clients.

Role & Responsibilities:

Define the overall data platform architecture (Lakehouse/EDW), including reference patterns (Medallion, Lambda, Kappa), technology selection, and integration blueprint.
Design conceptual, logical, and physical data models to support multi-tenant and vertical-specific data products; standardize logical layers (ingest/raw, staged/curated, serving).
Establish data governance, metadata, cataloging (e.g., Unity Catalog), lineage, data contracts, and classification practices to support analytics and ML use cases.
Define security and compliance controls: access management (RBAC/IAM), data masking, encryption (in transit/at rest), network segmentation, and audit policies.
Architect scalability, high availability, disaster recovery (RPO/RTO), and capacity & cost management strategies for cloud and hybrid deployments.
Lead selection and integration of platform components (Databricks, Delta Lake, Delta Live Tables, Fivetran, Azure Data Factory / Data Fabric, orchestration, monitoring/observability).
Design and enforce CI/CD patterns for data artifacts (notebooks, packages, infra-as-code), including testing, automated deployments and rollback strategies.
Define ingestion patterns (batch & streaming), file compacting/compaction strategies, partitioning schemes, and storage layout to optimize IO and costs.
Specify observability practices: metrics, SLAs, health dashboards, structured logging, tracing, and alerting for pipelines and jobs.
Act as technical authority and mentor for Data Engineering teams; perform architecture and code reviews for critical components.
Collaborate with stakeholders (Data Product Owners, Security, Infrastructure, BI, ML) to translate business requirements into technical solutions and roadmap.
Design, develop, test, and deploy processing modules using Spark (PySpark/Scala), Spark SQL, and database stored procedures where applicable.
Build and optimize data pipelines on Databricks and complementary engines (SQL Server, Azure SQL, AWS RDS/Aurora, PostgreSQL, Oracle).
Implement DevOps practices: infra-as-code, CI/CD pipelines (ingestion, transformation, tests, deployment), automated testing and version control.
Troubleshoot and resolve complex data quality, performance, and availability issues; recommend and implement continuous improvements.

Hard Skills - Must have:

Previous experience as architect or lead technical role on enterprise data platforms.
Hands-on experience with Databricks technologies (Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, Structured Streaming).
Strong expertise in Spark (PySpark and/or Scala), Spark SQL and distributed job optimization.
Solid background in data warehouse and lakehouse design; practical familiarity with Medallion/Lambda/Kappa patterns.
Experience integrating SaaS/ETL/connectors (e.g., Fivetran), orchestration platforms (Airflow, Azure Data Factory, Data Fabric) and ELT/ETL tooling.
Experience with relational and hybrid databases: MS SQL Server, PostgreSQL, Oracle, Azure SQL, AWS RDS/Aurora or equivalents.
Proficiency in CI/CD for data pipelines (Azure DevOps, GitHub Actions, Jenkins, or similar) and packaging/deployment of artifacts (.whl, containers).
Experience with batch and streaming processing, file compaction, partitioning strategies and storage tuning.
Good understanding of cloud security, IAM/RBAC, encryption, VPC/VNet concepts, and cloud networking.
Familiarity with observability and monitoring tools (Prometheus, Grafana, Datadog, native cloud monitoring, or equivalent).

Hard Skills - Nice to have/It's a plus:

Automation experience with CICD pipelines to support deployment and integration workflows including trunk-based development using automation services such as Azure DevOps, Jenkins, Octopus.
Advanced proficiency in Pyspark for advanced data processing tasks.
Advance proficiency in spark workflow optimization and orchestration using tools such as Asset Bundles or DAG (Directed Acyclic Graph) orchestration.
Certifications: Databricks Certified Data Engineer / Databricks Certified Professional Architect, cloud architect/data certifications (AWS/Azure/GCP).

Soft Skills / Business Specific Skills:

Ability to identify, troubleshoot, and resolve complex data issues effectively.
Strong teamwork, communication skills and intellectual curiosity to work collaboratively and effectively with cross-functional teams.
Commitment to delivering high-quality, accurate, and reliable data products solutions.
Willingness to embrace new tools, technologies, and methodologies.
Innovative thinker with a proactive approach to overcoming challenges.

At Allata, we value differences.

Allata is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees.

Allata makes employment decisions without regard to race, color, creed, religion, age, ancestry, national origin, veteran status, sex, sexual orientation, gender, gender identity, gender expression, marital status, disability or any other legally protected category.

This policy applies to all terms and conditions of employment, including but not limited to, recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.

About the Company

Allata

Strategy, architecture, and enterprise software development

Allata is a strategy, architecture and enterprise‑level application development firm that helps clients grow opportunities, improve efficiencies and automate processes through custom technologies. It defines vision and value creation, shapes strategic product designs, and manages enterprise delivery to enable business agility at the intersection of people, process, and technology. The company differentiates itself with a networked team culture that prioritizes caring, client focus, and responsible growth—including a monthly charitable donation equal to 2% of net income—and by delivering transformational work across industries. Allata’s goal is to help clients become more nimble, transformative, and disruptive while building a connected team of talented professionals who can stand strong on their own yet work together to achieve client goals.

Company size: 201-500
Company stage: Not listed
Total funding: Not listed
Headquarters: Dallas, TX
Founded: 2016

Recently Posted Jobs

Explore more opportunities at Allata

See more jobs