Big Data / PySpark Engineering Lead
No salary listed
Expert
Pune, Maharashtra, India
In Person
Data & AnalyticsData Engineering
About the job
Requirements
- At least 12 years of experience building software and in platform engineering.
- Experience in data engineering focused on big data ecosystems.
- Knowledge of Hadoop, YARN, Hive, Impala, Spark, and Spark SQL, including extensive high-volume data-processing pipeline development.
- Expert-level programming proficiency and hands-on experience in Python.
- Familiarity with Avro, Parquet, CSV, and JSON data formats.
- Hands-on experience writing SQL queries.
- Extensive experience with Unix-based operating systems and shell scripting.
- Experience with source code management tools such as Bitbucket and Git.
- Hands-on proficiency with Hadoop, Spark, Hive, Kafka, and NoSQL databases such as MongoDB and HBase.
- Experience with query engines such as Trino, Presto, and Starburst.
- Computer science fundamentals in data structures, algorithms, databases, and operating systems.
- Ability to reverse engineer and read legacy SQL or scripts and document the business logic before migration.
- Experience with data lineage tools such as Collibra or Informatica.
- Experience managing the technical impact on the business of switching from legacy business intelligence tools to modern query engines such as Starburst.
Responsibilities
- Design and implement scalable, fault-tolerant batch and real-time data-processing pipelines.
- Develop robust data models and schema designs optimized for performance and storage efficiency.
- Evaluate and integrate emerging tools and frameworks, such as Spark, Flink, and Kafka, into the existing stack.
- Analyze complex issues and develop innovative solutions using interpretive thinking.
- Develop knowledge of how business areas, including architecture and infrastructure, integrate to accomplish business goals.
- Lead strategic migration of data and logic from legacy platforms, such as on-premises SQL Servers, to a modern Data Lakehouse environment.
- Re-engineer stored procedures and complex legacy ETL jobs into scalable distributed processing frameworks using Spark with Python and Starburst or Trino.
- Design and implement automated data parity testing frameworks to verify accuracy and consistency between legacy outputs and new big data results.
- Map and transform legacy relational schemas into flexible, high-performance cloud formats such as Parquet, Avro, or Iceberg.
- Orchestrate phased migration strategies, including parallel runs and shadow execution, to ensure uninterrupted service for downstream business applications and reporting tools.
- Establish performance baselines on legacy systems and ensure the new big data architecture meets or exceeds those benchmarks at scale.
- Resolve high-impact problems and projects by evaluating complex business processes, system processes, and industry standards.
- Write clean, high-performance Python code.
- Optimize complex SQL queries and tune distributed computing clusters to reduce latency and costs.
- Implement validation and encryption standards to protect data integrity and security.
- Build and maintain continuous integration and continuous deployment pipelines for automated testing and deployment of data jobs.
- Monitor system health and troubleshoot performance bottlenecks across the data lifecycle.
- Provide technical mentorship and code reviews for junior and mid-level engineers, coach mid-level developers and analysts, and allocate work as necessary.
- Translate complex business requirements into technical specifications.
- Collaborate with Product Managers to ensure data availability for downstream analytics, business models, and users.
- Assess risk in business decisions, consider the firm's reputation and the safeguarding of the firm, its clients, and assets, drive compliance with applicable laws, rules, and regulations, adhere to policy, apply sound ethical judgment to personal behavior, conduct, and business practices, and transparently escalate, manage, and report control issues.
- Partner with multiple management teams to integrate functions toward shared goals and identify and define system enhancements for new products and process improvements.
Desired Qualifications
- Identify root causes of problems and prevent recurrence rather than only fixing bugs.
- Explain the rationale behind technical decisions to non-technical stakeholders.
- Use automation and AI tools to expedite delivery, with familiarity with AI tools.
About the company
Citi runs banking, investment and payment services for companies, governments and individuals around the world. Its global network connects institutions with local banking systems, while its US consumer business sells credit cards and deposit accounts. Corporate clients use Citi to manage cash, finance trade and raise capital, and investors use its trading, custody and wealth services to hold and manage assets. The bank earns interest on loans and fees from payments, securities transactions, financial advice and account services. Citi traces its banking roots to 1812; the Citigroup holding company was formed in 1998, and its headquarters are in New York City.
Company Size
10,001+
Company Stage
IPO
Headquarters
New York City, New York
Founded
1812
Get referred to Citi
See people who can refer or advise you
Simplify's Take
What believers are saying
- Revenue is expanding: Citi reported $24.8 billion in the June 2026 quarter, up 14% year over year.
- US parents get protected time: eligible Citi employees receive 16 weeks of bonding leave at full base pay.
- Institutional deposits are sticky: Citi says around 80% of Services deposits come from clients with relationships exceeding 15 years.
What critics are saying
- Lending losses remain substantial: Citi's net credit losses reached $2.4 billion in the June 2026 quarter.
- Australian hybrid roles require office time: Citi sets a minimum of three office days weekly for designated hybrid positions.
- Job cuts carry a cost: Citi booked roughly $500 million in severance in the March 2026 quarter.
What makes Citi unique
- Corporate payments move at scale: Citi processes around $6 trillion daily through its Treasury and Trade business.
- Deposits move beyond banking hours: Citi Token Services transfers cash between participating Citi branches around the clock.
- Employees use a shared AI workspace: Citi's Stylus platform reaches over 180,000 people for research and multi-step workflows.
Help us improve and share your feedback! Did you find this helpful?
Benefits
- Health Insurance
- Dental Insurance
- Vision Insurance
- Life Insurance
- Disability Insurance
- 401(k) Retirement Plan
- 401(k) Company Match
- Wellness Program
- Paid Vacation
- Paid Sick Leave
- Paid Holidays