A

Architecture in Motion

Designs information flows and operating models

Senior Data Engineer - AI & LLM Solutions

Data EngineeringPosted on Nov 17
Full-timeNo salary listed

About the Job

Requirements

  • A bachelor's degree in Computer Science, Software Engineering, Information Technology, Data Science, or a related discipline.
  • At least 8 years of professional experience in Data Engineering, Data Platforms, or Cloud Data Solutions.
  • At least 5 years of hands-on experience with Microsoft Azure Data Services.
  • Proven experience delivering enterprise-scale data, analytics, and AI solutions.
  • Strong experience with modern cloud data platforms and data architectures.
  • Experience working in consulting, professional services, or enterprise environments.
  • Strong analytical and problem-solving skills.
  • Ability to design and implement scalable data solutions.
  • Strong understanding of enterprise data architecture.
  • Ability to work independently in a remote, cross-functional environment.
  • Strong communication and stakeholder management skills.
  • Ability to mentor engineers and provide technical leadership.
  • Focus on security, quality, performance, and reliability.
  • Ability to translate business requirements into technical solutions.
  • Experience with Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage Gen2, Azure Databricks, Azure Event Hubs, data warehouse and lakehouse architecture, and medallion architecture.
  • Experience with Python, SQL, PySpark, Spark SQL, Kusto Query Language, REST API integrations, and ETL/ELT development.
  • Experience with SQL Server, PostgreSQL, MySQL, NoSQL databases, data modeling, and optimization.
  • Experience with Azure OpenAI, generative AI, prompt engineering, Retrieval-Augmented Generation, vector databases or vector search, machine-learning pipelines, natural language processing fundamentals, LLM integration patterns, and AI-ready data architecture.
  • Experience with Power BI, Tableau, semantic models, executive reporting, and KPI frameworks.
  • Experience with Microsoft Purview, data governance, data classification, data lineage, data cataloging, access control, and enterprise security standards; awareness of SOC 2, NIST, and CMMC.
  • Experience with Azure DevOps, Git, CI/CD pipelines, infrastructure-as-code concepts, containerization concepts, and cloud resource optimization.

Responsibilities

  • Design and develop enterprise-scale data solutions using Microsoft Fabric, Azure Data Factory, Azure Synapse Analytics, Azure Data Lake Storage, and Azure Databricks.
  • Build and optimize scalable ETL/ELT pipelines for structured, semi-structured, and unstructured data.
  • Develop modern lakehouse, data warehouse, and medallion architecture solutions.
  • Design and implement batch and real-time data processing frameworks.
  • Lead data modernization, migration, and integration initiatives from legacy platforms.
  • Develop data foundations that support artificial intelligence, machine learning, and generative AI solutions.
  • Prepare, transform, and optimize datasets for AI applications, model training, and fine-tuning.
  • Design and implement Retrieval-Augmented Generation architectures and vector search solutions.
  • Support Azure OpenAI, Microsoft Copilot, and enterprise LLM implementations.
  • Develop data pipelines and architectures that enable AI-driven applications.
  • Collaborate with data scientists and AI engineers to operationalize AI and machine-learning solutions.
  • Build distributed data-processing solutions using PySpark and Spark SQL.
  • Develop automated data-integration frameworks connecting APIs, SaaS applications, databases, and cloud services.
  • Design reusable data-engineering frameworks, components, and standards.
  • Optimize large-scale workloads for performance, scalability, reliability, and cost efficiency.
  • Implement data-quality, validation, monitoring, and error-handling processes.
  • Implement data-governance and cataloging solutions using Microsoft Purview and related technologies.
  • Establish and maintain data lineage, classification, cataloging, and access controls.
  • Ensure data platforms align with enterprise security, privacy, and compliance requirements.
  • Support governance and security requirements across SOC 2, NIST, and CMMC frameworks.
  • Implement secure data-access and protection practices across cloud data platforms.
  • Design and support semantic data models for Power BI and enterprise analytics.
  • Enable self-service analytics while maintaining appropriate governance and data-quality standards.
  • Support KPI frameworks, executive dashboards, reporting, and predictive-analytics initiatives.
  • Collaborate with analytics teams to ensure reliable, high-quality data availability.
  • Provide technical guidance and mentorship to junior and mid-level data engineers.
  • Participate in solution architecture, technical design reviews, and implementation planning.
  • Work directly with clients and stakeholders to understand requirements and translate business needs into scalable technical solutions.
  • Collaborate with AI engineers, cloud engineers, solution architects, data scientists, and business analysts.
  • Participate in technical discussions and presales activities where required.
  • Develop technical documentation, engineering standards, and reusable implementation patterns.
  • Troubleshoot and resolve data-pipeline, platform, and integration issues.
  • Monitor and optimize data workloads and cloud resources.
  • Support enhancements, upgrades, and ongoing maintenance of data platforms.
  • Identify opportunities to improve data quality, automation, performance, and operational efficiency.
  • Stay current with emerging technologies in Microsoft Fabric, Azure, AI, generative AI, and data engineering.

Desired Qualifications

  • Microsoft Certified: Fabric Analytics Engineer Associate certification.
  • Microsoft Certified: Azure Data Engineer Associate certification.
  • Microsoft Certified: Azure Data Scientist Associate certification.
  • Microsoft Certified: Azure Solutions Architect Expert certification.
  • Databricks Certified Data Engineer certification.
  • Relevant AI, machine-learning, or generative-AI certifications.

About the Company

Architecture in Motion

Designs information flows and operating models

Architecture in Motion helps organizations plan and execute digital transformation at scale. It provides AIM, a framework to design and validate business information flows and the operating model, and uses that AIM as a shared business model to plan and execute projects and manage the portfolio. By standardizing how information moves and how work is organized, it enables consistent project planning, execution, and portfolio management across the enterprise. Its goal is to help companies implement large-scale digital changes in a coordinated, manageable way that aligns projects to a common model.

Company size
1-10
Company stage
Not listed
Total funding
Not listed
Headquarters
Sydney NSW, Australia
Founded
2016

Recently Posted Jobs

Explore more opportunities at Architecture in Motion

See more jobs