Data Engineer II

Description Summary: ​​M3 is seeking a skilled and experienced Data Engineer to join our team. The Data Engineer will be responsible for designing, developing, and maintaining data infrastructure, storage solutions, and pipelines. The ideal candidate will possess expertise in data management, security, and optimization, along with proficiency in ETL/ELT, batch and near real-time workloads, and CI/CD practices. Collaboration with cross-functional teams and effective communication are essential for success in this role.​​

Essential Duties:

​​The duties listed below are the essential functions of this position, and they may change as the needs of the company demand. All associates are expected to do what is necessary to get the work done and to cooperate fully with their supervisor’s requests for additional or altered duties.​

• Develop and maintain data infrastructure, including databases, data warehouses, and data lakes, to support business needs.
• Design and build scalable and efficient data storage solutions to accommodate large volumes of structured and unstructured data.
• Design, develop, and deploy data pipelines to ingest, process, and transform data from various sources.
• Implement data security measures, including encryption, access controls, and auditing, to protect sensitive data.
• Manage data quality by implementing monitoring and validation processes to ensure accuracy, completeness, and consistency.
• Optimize data performance by tuning queries, optimizing storage, and implementing caching and indexing strategies.
• Develop and maintain data models to organize and structure data for efficient storage, retrieval, and analysis.
• Integrate data from multiple sources, including internal databases, external APIs, and third-party data providers, to support analytical and reporting needs.
• Utilize Microsoft Azure DevOps Repos and Release Pipelines and GitHub for version control, collaboration, and deployment of data solutions.
• Communicate effectively with stakeholders to understand data requirements and provide technical expertise and support.
• Other duties as assigned.  

Education/Training/Experience: 

• 3+ years of experience as a Data Engineer or similar role.
• Completion of a ​Bachelors Degree​ is  preferred.  Strong preference for the completion of a major in ​IT, MIS or Computer Science​ ; or an equivalent combination of education and experience.
• Strong proficiency in data infrastructure and data management.
• Strong proficiency in SQL for querying and manipulating data.
• Experience with ETL and ELT for data pipeline development.
• Experience in designing and implementing both batch processing workloads.
• Experience in designing and implementing near real-time streaming workloads.
• Proficiency in Python for data analysis, manipulation, and automation is a plus.
• Experience with CI/CD practices and tools for automated deployment.
• Familiarity with Microsoft Azure DevOps Repos and Release Pipelines and GitHub for version control and collaboration.
• Knowledge of cloud data lake services and data management best practices.
• Excellent communication and collaboration skills.  

Physical Requirements:

• Ability to sit and/or stand for extended periods.
• Ability to perform work on a computer for extended periods. 
• Ability to attend work per assigned schedule and attend meetings with excellent attendance and punctuality.
• Ability to bend and lift up to 25 lbs.