Technical Architect

Responsibilities

• Lead the design and implementation of a robust, scalable, and reusable data ingestion framework using Microsoft Fabric
• Building a metadata-driven solution for ingesting data from CSV sources into the Data Lake
• Work closely with stakeholders to understand data sources, ensure compliance, and guide development teams to deliver efficient and adaptable data ingestion pipelines.

Must Have:

• 8-10 Yrs of Experience in Building data architecture using various data engineering techniques
• In-depth knowledge and hands-on experience with Microsoft Fabric components, including Lakehouse, One Lake, Delta Lake format, Fabric Pipelines, Dataflows Gen2, and Spark notebooks/Synapse Data Engineering
• Strong understanding of data lake concepts, including optimal use of One Lake for data storage, Delta Lake
• Experience in designing and implementing metadata-driven solutions for data ingestion
• Expertise in various data ingestion patterns, including batch processing (specifically for CSV files)
• Proficient in data modeling standards, potentially including Medallion architecture (Bronze, Silver, Gold layers)
• Strong proficiency in PySpark for data processing and transformation within Fabric Spark environments
• Advanced SQL skills for querying data, managing database interactions
• Strong analytical and problem-solving skills to assess data requirements, troubleshooting issues

Good to Have: 

• Proficiency in PySpark for data processing and transformation within Fabric Spark environments 
• Understanding of data governance principles, data security measures
• Knowledge of monitoring data pipeline performance, troubleshooting issues