U

University of Texas at Austin

Public research university in Austin, TX

Lead Data Engineer

Data EngineeringConfirmed live in the last 24 hours
Full-timeTexas, USA$125k - $143.7k

About the Job

Requirements

  • A Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent professional experience.
  • At least 5 years of experience designing, implementing, and optimizing complex production-grade data pipelines or enterprise-scale data platforms.
  • At least 5 years of experience in cloud-based data engineering using Databricks and Amazon Web Services, such as Glue, S3, Lambda, and Redshift.
  • At least 3 years of experience managing or leading teams of data and/or software engineers, including mentorship, performance management, and project delivery.
  • Expertise in Python, PySpark, and SQL, with a strong understanding of data modeling, stored procedures, and scalable data transformations.
  • Proven experience architecting and implementing ETL/ELT solutions across relational, non-relational, and lakehouse environments, such as Delta Lake, Parquet, or Iceberg.
  • Experience designing and managing CI/CD pipelines and infrastructure as code using tools such as Databricks Repos, CDK, Terraform, or GitHub Actions.
  • Knowledge of test-driven development and data quality frameworks to ensure reliability and reproducibility across data workflows.
  • Deep understanding of data governance, security, and compliance standards in cloud environments.
  • Analytical, problem-solving, and debugging ability across distributed data systems.
  • Ability to communicate complex technical concepts clearly to technical and non-technical audiences.
  • Experience supervising, mentoring, and guiding junior team members on technical and professional development.

Responsibilities

  • Design, architect, and deliver production-grade, scalable data pipelines and AI-ready data platforms using Databricks, AWS cloud-native services, and modern data engineering frameworks.
  • Lead end-to-end implementation of lakehouse data pipelines, ensuring performance, reliability, and cost efficiency.
  • Champion industry best practices for data engineering.
  • Conduct and participate in peer code reviews to maintain code quality and consistency across the team.
  • Identify and resolve bottlenecks in data ingestion, transformation, and orchestration using Databricks Delta Live Tables, Spark optimization techniques, and workflow automation.
  • Implement systems for data quality, observability, governance, and compliance using tools such as Unity Catalog, Delta Lake, and data validation frameworks.
  • Lead technical knowledge-sharing sessions on AI/ML integration, data lakehouse architecture, and emerging data technologies.
  • Define project milestones, timelines, and deliverables for data and AI initiatives, ensuring timely, high-quality outcomes.
  • Collaborate with internal and external stakeholders, including data architects, system architects, business users, Agile team members, and other D2I groups.
  • Manage project priorities, sprint planning, and team workloads while balancing innovation with delivery.
  • Communicate risks, dependencies, and resource constraints, and develop mitigation plans for on-time project delivery.
  • Supervise and mentor 2–5 Data Engineers working on cloud, Databricks, and AI pipeline initiatives.
  • Foster continuous learning, experimentation, and technical excellence, encouraging engineers to explore AI and automation use cases.
  • Participate in recruiting, onboarding, and developing data engineering talent with Databricks and AI skillsets.
  • Conduct performance reviews, set development goals, and create individualized growth plans for team members.
  • Encourage collaboration across Data, AI/ML, Analytics, and Infrastructure teams.
  • Provide regular updates on project progress, technical challenges, and milestones to technical and business stakeholders.
  • Translate technical concepts related to Databricks, AI, and data architecture into clear narratives for non-technical audiences.
  • Foster a transparent communication culture and provide actionable feedback to promote a growth mindset.
  • Document data engineering processes, architectures, and standards for reuse, governance, and knowledge continuity.
  • Stay current with advancements in AI, data engineering, and the Databricks ecosystem, evaluating new tools and frameworks for adoption.
  • Pilot and promote solutions such as AI-assisted data quality checks, data observability automation, and intelligent pipeline optimization.
  • Perform other duties as assigned, contributing to the organization’s data-driven and AI-enabled transformation.

Desired Qualifications

  • At least 8 years of experience in Data Engineering or related fields, including at least 5 years of hands-on experience building and optimizing data pipelines on Databricks or similar large-scale data platforms.
  • Experience implementing lakehouse architectures using Databricks Delta Lake, Delta Live Tables, and Unity Catalog for governance and scalability.
  • Experience designing AI-ready data platforms and integrating machine learning pipelines using tools such as MLflow or model registry frameworks.
  • At least 3 years of experience managing or leading cross-functional technical teams and fostering collaboration between Data Engineering, Analytics, and AI/ML teams.
  • At least 5 years of experience with Agile software development methodologies and project tracking systems such as JIRA.
  • Expertise in distributed data processing and streaming frameworks such as Apache Spark, Kafka, Flink, or Airflow for orchestration and automation.
  • Familiarity with data observability, cost optimization, and performance tuning in Databricks and cloud-native architecture.
  • Professional certifications such as Databricks Certified Data Engineer Professional, AWS Solutions Architect, or AWS Data Analytics Specialty.
  • Ability to introduce new technologies and best practices to modernize data environments and promote AI/analytics maturity.
  • Interest in continuous learning and staying current with emerging technologies in data engineering, AI integration, and the Databricks ecosystem.

About the Company

University of Texas at Austin

Public research university in Austin, TX

University of Texas at Austin is a public research university in Austin, Texas. It offers undergraduate, graduate, and professional education across a broad range of fields.

Company size
Not listed
Company stage
Not listed
Total funding
Not listed
Headquarters
Not listed
Founded
Not listed

Recently Posted Jobs

Explore more opportunities at University of Texas at Austin

See more jobs