Azure Data Engineer

Job Description

The Azure Data Engineer is responsible for designing, implementing, and managing robust data solutions to meet business needs. This includes overseeing data pipelines, Azure data platform services, and delivering actionable insights through Power BI dashboards and reports. The specialist will play a critical role in transforming raw data into valuable insights, ensuring data accuracy, and supporting business intelligence initiatives across the organization.

1. Azure Data Platform Management: Design, implement, and manage Azure SQL databases, Azure Data Lake Storage, and related Azure & Fabric services.
2. ETL Processes and Data Pipelines: Create, monitor, and optimize ETL (Extract, Transform, Load) workflows to integrate data from multiple sources into centralized systems, ensuring reliability and performance.
3. SQL and Python Development: Utilize SQL for querying, analysing, and managing relational databases, and develop Python scripts to automate data processes, integrate data sources, and support advanced analytics.
4. Big Data and Advanced Analytics: Leverage Azure Databricks for big data processing, analytics, and machine learning workflows, and utilize tools like Apache Spark and Hadoop for processing large datasets.
5. Data Warehousing: Develop and maintain enterprise data warehouses to support analytics and reporting needs while ensuring seamless integration with visualization tools.
6. Data Security and Compliance: Implement robust data security measures to protect sensitive data, maintain compliance with organizational and regulatory standards, and support disaster recovery processes.
7. Collaboration and Documentation: Collaborate with cross-functional teams to gather requirements and deliver data solutions while maintaining detailed documentation of data architectures, ETL processes, and Power BI standards.
8. AI Enablement: Design, implement and Manage data solutions for the AI needs.

IT Policies and IT Processes impacted/addressed by this position:

1. Database Management Process: Adherence to data governance processes and policies to ensure data accuracy and security.
2. Data Security Policy: Ensuring compliance with data protection and security guidelines.
3. Change Management Policy: Following procedures for managing changes in technology and processes.
4. Business Continuity Policy: Supporting business continuity planning and disaster recovery processes.
5. IT Incident Management: Participating in the resolution of IT incidents related to data solutions.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Business Administration, or a related field.

• Advanced certifications in Azure Data Engineering, SQL, or Microsoft Power BI are preferred.

Additional Information

Experience:

• Minimum of 7 years of experience in data engineering, data warehousing, ETL processes, and business intelligence development.
• Proven expertise in Azure Data Factory, Azure Databricks, and Power BI development.
• Strong experience in SQL for database management, querying, and optimization.
• Any POC project for AI enablement for end users.
• Demonstrated proficiency in Python for data automation, analytics, and advanced transformations.
• Experience in managing data pipelines and integrating large data sets for analysis.
• Demonstrated experience in collaborating with stakeholders and presenting data-driven insights.
• Experience in building Data solutions for AI recipients that successfully helped the business with a tangible benefit.

Skills & Abilities:

• Proficient in Azure services: Data Factory, Databricks, Data Lake, SQL Database, and Synapse Analytics.
• Advanced Power BI expertise, including DAX, data modelling, and report development.
• Strong SQL and Python skills for data processing, analysis, and automation.
• Experience with big data frameworks like Apache Spark and Hadoop.
• Proficiency in relational databases: Microsoft SQL Server, MySQL, and PostgreSQL.
• Excellent organizational, problem-solving, and communication skills.
• Familiarity with data security and compliance standards.