Senior Lead Data Engineer

About the Role:

Grade Level (for internal use):

12

We are seeking a hands-on Senior Lead Data Engineer / Data Platform Lead to support our offshore team in India. This role focuses on executing data pipeline onboarding, migration, and support activities within our enterprise data platform. You will work closely with the Databricks team and internal stakeholders to ensure smooth transition, ongoing refinement, and reliable operation of data pipelines and integrations in a Databricks-on-AWS lakehouse environment. Databricks on AWS positions data engineering around governed ingestion, transformation, and delivery of high-quality data for analytics and downstream use cases. 

This is a delivery-focused role requiring strong technical expertise in data engineering, with responsibilities for supporting data onboarding, pipeline maintenance, platform support, and cloud-native operational excellence. You will lead a small team of engineers, providing technical guidance and ensuring high-quality execution across scalable, secure, and governed data workflows leveraging Amazon S3, AWS Glue, AWS IAM, and AWS Lake Formation alongside Databricks capabilities such as Delta Lake, Unity Catalog, and open table format interoperability where appropriate.

Key Responsibilities

Data Pipeline Support & Migration

• Assist in the migration of existing data pipelines to the Databricks platform, following established patterns and runbooks.
• Support the ongoing operation, troubleshooting, and optimization of pipelines post-migration.
• Implement and follow engineering best practices for data ingestion, transformation, and monitoring.
• Ensure pipelines are reliable, performant, and maintainable.
• Support data ingestion and storage patterns using Amazon S3 as the core cloud object store for lakehouse data layers, with Databricks processing and transformation built on top of that storage foundation. 
• Contribute to migration and optimization activities involving AWS Glue for metadata cataloging, schema discovery, or ETL interoperability where required across the enterprise AWS data ecosystem.
• Support analytical integration patterns with downstream platforms such as Amazon Athena when business consumers require governed access beyond Databricks-native consumption paths. 

Data Onboarding & Asset-Agnostic Support

• Support offshore teams in onboarding new data assets to the enterprise platform using standardized, asset-agnostic processes.
• Help teams prepare, validate, and publish data assets, ensuring consistency and compliance with platform standards.
• Document onboarding procedures and assist in resolving onboarding issues.
• Help standardize onboarding into Bronze, Silver, and Gold data layers and support practical lakehouse patterns for ingestion, curation, and consumption in Databricks on AWS. 
• Support metadata-driven onboarding workflows using AWS Glue Data Catalog and Databricks governance capabilities to improve discoverability, schema consistency, and reusable ingestion patterns.
• Assist with secure data onboarding by aligning access permissions and environment controls through AWS IAM and AWS Lake Formation, alongside Unity Catalog permissions and governance policies for managed access to datasets. 

Platform Integration Support

• Support integration of data pipelines with the enterprise data mastering platform.
• Assist in data quality checks, metadata management, and reconciliation activities.
• Help troubleshoot and resolve issues related to data ingestion and mastering workflows.
• Support interoperability across AWS and Databricks services for ingestion, transformation, and governed publishing of mastered or standardized datasets.
• Contribute to table design and data publication patterns using Delta Lake and, where applicable, Apache Iceberg to support interoperable lakehouse consumption models.
• Assist with auditability and lineage-oriented controls through metadata, access management, and governed publishing standards.

Team Collaboration & Delivery

• Work closely with the offshore team to prioritize tasks, meet delivery deadlines, and maintain quality.
• Provide technical guidance, review code, and mentor junior team members.
• Communicate progress, risks, and issues effectively to stakeholders.
• Collaborate with platform, governance, and cloud teams to align engineering delivery with enterprise AWS standards, security controls, and support processes.
• Promote practical engineering discipline across source control, peer review, CI/CD, release management, and environment consistency.

Operational Excellence

• Follow established operational procedures for deployment, monitoring, and incident management.
• Support production support activities, including issue resolution and performance tuning.
• Contribute to documentation and knowledge sharing.
• Support cloud-native monitoring and operational visibility using tools such as Amazon CloudWatch, logging, alerting, and job observability for production data pipelines.
• Contribute to deployment automation and release reliability using CI/CD, infrastructure as code, and cloud-native orchestration patterns, including AWS services such as AWS Step Functions, AWS Lambda, or equivalent enterprise tooling where applicable.
• Help maintain secure and compliant platform operations through access control, secrets handling, environment segregation, and operational readiness practices.

Required Qualifications

• 8+ years of experience in data engineering, pipeline development, or related roles.
• Proven experience supporting data onboarding, migration, and pipeline operations.
• Hands-on expertise with data engineering tools such as Spark, SQL, Python, or similar.
• Experience working with Databricks or similar cloud data platforms.
• Hands-on experience with relevant AWS data engineering services, especially Amazon S3, AWS Glue, and core cloud security concepts such as AWS IAM; familiarity with AWS Lake Formation, Amazon Athena, AWS Lambda, or Amazon Kinesis is strongly preferred for enterprise-scale data platform support. Industry-standard AWS data engineering stacks commonly center on these services for storage, ETL, governance, analytics, and streaming. 
• Familiarity with data quality, metadata, lineage, and basic data governance concepts.
• Working knowledge of Delta Lake, Databricks Unity Catalog, and exposure to Apache Iceberg or similar open table/storage formats for lakehouse architectures. 
• Understanding of secure data access, governance, and platform controls across Databricks and AWS environments.
• Familiarity with software engineering best practices, including version control, automated testing, CI/CD, infrastructure as code, and monitoring.
• Strong problem-solving skills and ability to troubleshoot production issues.
• Good communication skills and ability to collaborate with global teams.

Preferred Qualifications

• Experience supporting enterprise data onboarding and integration workflows.
• Knowledge of Delta Lake, Apache Iceberg, or similar storage formats.
• Exposure to data mastering or data governance frameworks.
• Experience with AWS Lake Formation, AWS Glue Data Catalog, Amazon Athena, Amazon Kinesis, or AWS Step Functions in support of modern cloud data platforms.
• Experience supporting Databricks on AWS lakehouse implementations, including governed data access with Unity Catalog and cloud storage patterns built on Amazon S3. 
• Proficiency in Python and SQL, with experience in PySpark or similar distributed processing frameworks.
• Familiarity with infrastructure as code and cloud deployment automation.
• Certifications in Databricks, AWS, or related data platforms.

What’s In It For You?

Our Mission:

Advancing Essential Intelligence.

Our People:

We're more than 35,000 strong worldwide—so we're able to understand nuances while having a broad perspective. Our team is driven by curiosity and a shared belief that Essential Intelligence can help build a more prosperous future for us all.From finding new ways to measure sustainability to analyzing energy transition across the supply chain to building workflow solutions that make it easy to tap into insight and apply it. We are changing the way people see things and empowering them to make an impact on the world we live in. We’re committed to a more equitable future and to helping our customers find new, sustainable ways of doing business. Join us and help create the critical insights that truly make a difference.

Our Values:

Integrity, Discovery, Partnership

Throughout our history, the world's leading organizations have relied on us for the Essential Intelligence they need to make confident decisions about the road ahead. We start with a foundation of integrity in all we do, bring a spirit of discovery to our work, and collaborate in close partnership with each other and our customers to achieve shared goals.

Benefits:

We take care of you, so you can take care of business. We care about our people. That’s why we provide everything you—and your career—need to thrive at S&P Global.

Our benefits include: 

• Health & Wellness: Health care coverage designed for the mind and body.

• Flexible Downtime: Generous time off helps keep you energized for your time on.

• Continuous Learning: Access a wealth of resources to grow your career and learn valuable new skills.

• Invest in Your Future: Secure your financial future through competitive pay, retirement planning, a continuing education program with a company-matched student loan contribution, and financial wellness programs.

• Family Friendly Perks: It’s not just about you. S&P Global has perks for your partners and little ones, too, with some best-in class benefits for families.

• Beyond the Basics: From retail discounts to referral incentive awards—small perks can make a big difference.

For more information on benefits by country visit: https://spgbenefits.com/benefit-summaries

Global Hiring and Opportunity at S&P Global:

At S&P Global, we are committed to fostering a connected and engaged workplace where all individuals have access to opportunities based on their skills, experience, and contributions. Our hiring practices emphasize fairness, transparency, and merit, ensuring that we attract and retain top talent. By valuing different perspectives and promoting a culture of respect and collaboration, we drive innovation and power global markets.

Recruitment Fraud Alert:

If you receive an email from a spglobalind.com domain or any other regionally based domains, it is a scam and should be reported to [email protected]. S&P Global never requires any candidate to pay money for job applications, interviews, offer letters, “pre-employment training” or for equipment/delivery of equipment. Stay informed and protect yourself from recruitment fraud by reviewing our guidelines, fraudulent domains, and how to report suspicious activity here.

20 - Professional (EEO-2 Job Categories-United States of America), IFTECH202.2 - Middle Professional Tier II (EEO Job Group), SWP Priority – Ratings - (Strategic Workforce Planning)