Senior Data Engineer

Federal Requirements:  

• Ability to obtain a U.S. Federal Position of Trust clearance designation.  
• Must reside in and be able to perform work in the United States.  
• Must have lived in the United States for 3 of the last 5 years.  

Required Qualifications:

• Master’s degree in computer science, Data Engineering, or a related field with a minimum of 4 years of experience in data engineering (PhD is a plus). 
• At least 5 years of experience in programming with Python, focusing on data engineering tasks and scripting. 
• A minimum of 3 years of hands-on experience with Apache Spark for large-scale data processing, including building data visualizations using PySpark and Jupyter Notebooks. 
• Proficiency in data manipulation and analysis using Python libraries such as NumPy and Pandas. 
• Proven expertise in designing and managing data pipelines using AWS services, including AWS EMR and AWS S3. 
• 4+ years of experience working with relational databases and AWS Redshift, with a strong understanding of SQL for data manipulation and querying. 
• At least 2 years of experience utilizing Jupyter Notebooks for data exploration, analysis, visualization, and collaboration. 
• Strong knowledge of handling various data formats, including CSVs and Parquet files. 
• Extensive experience with cloud infrastructure, specifically AWS, and a thorough understanding of its services and capabilities. 

Preferred Qualifications:

• 2+ years of experience in Scala programming. 
• Familiarity with EMR Studios and Anaconda for data engineering and analytics. 
• Experience working with AWS Bedrock and other LLM SaaS platforms (such as OpenAI or similar) for AI/ML projects. 
• Proven experience in curating datasets for use in DAGs or model training processes. 
• Background in data engineering within the healthcare domain, particularly with administrative claims data or health insurance claims data. 
• Knowledge of CMS (Center for Medicare and Medicaid Services) protocols and experience with CMS’s Integrated Data Repository (IDR). 
• Understanding of the Hadoop ecosystem and distributed computing concepts. 

Responsibilities:

• Develop, implement, and optimize scalable data pipelines on AWS to ensure efficient processing and storage of large datasets. 
• Collaborate with cross-functional teams to define data requirements and establish effective data governance frameworks. 
• Design and maintain robust data infrastructure to support diverse data workloads, including batch processing using AWS EMR. 
• Manage large volumes of structured and unstructured data, utilizing AWS S3 and AWS Redshift for efficient storage and querying. 
• Create, maintain, and enhance data ingestion processes using Apache Spark to ensure data quality, integrity, and consistency. 
• Utilize PySpark and Jupyter Notebooks to build data visualizations that support data-driven decision-making and provide insights. 
• Work closely with stakeholders to understand data needs and develop innovative solutions that drive data-driven decision-making. 
• Utilize Jupyter Notebooks for data exploration, visualization, and sharing of insights to support data science and analytics efforts. 
• Implement and enforce best practices for data security and compliance, particularly when handling sensitive healthcare data. 

Benefits and Perks:

• 65%-75% company-sponsored (including dependents) premiums towards medical, dental and vision insurance. For eligible plans and tiers, we provide 100% company-paid medical insurance. 100% employer sponsored STD, LTD and life insurance (min $100K). Voluntary life insurance option available. 
• Retirement 401(k) plan with employer matching. Immediate vesting. 
• Vacation and sick leave. 
• Maternity and parental leave.  
• Discretionary bonuses, spot awards, gifts, and tenure-based rewards. 
• Company-sponsored role-based training and certifications.  
• Monthly DoordashDashPass subscription. 
• Group discounts via LifeMart ADP