Unison Group

Builds financial risk and compliance systems

Senior Data Engineer - Databricks and PySpark

Data EngineeringUpdated on Nov 11

Full-time

About the Job

Requirements

At least 5 years of data engineering experience.
Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, and Python.
Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents such as S3 and Lambda.
Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
Knowledge of ETL/ELT concepts, Lakehouse/Medallion architecture, and distributed processing.
Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
Ability to optimize Spark workloads using partitioning, caching, Z-ordering, and performance tuning.
Mandarin language proficiency is required.

Responsibilities

Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
Optimize Spark jobs for performance, reliability, and cost efficiency.
Implement data validation, quality checks, and monitoring with automated alerts and retries.
Design secure and governed datasets using Unity Catalog and cloud security best practices.
Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
Provide technical leadership and guidance to junior team members.

Desired Qualifications

Exposure to Oil & Gas or trading analytics, including SPARTA, KPLER, IIR, or OPEC.
Knowledge of Power BI or data visualization concepts.
Familiarity with Terraform, Scala, or PostgreSQL.
Experience with SharePoint development or .NET.

Senior Data Engineer (Databricks and Pyspark)-Mandarin Speaker

Unison Group · Kuala Lumpur, Malaysia · Full-time · Posted 2026-11-05

Salary: unknown currency 15,000–25,000

Workplace: on_site

Description

Position : Senior Data Engineer.
Location : Malaysia and Singapore.
Language : Mandarin Language must.
Role : Full Time Perm role + one month Bonus.

Requirements

We are looking for a Senior Data Engineer with strong expertise in Databricks, PySpark, Delta Lake, and cloud-based data pipelines. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.

Key Responsibilities

Design, build, and optimize data ingestion and transformation pipelines using Databricks, PySpark, and Python.
Implement Delta Lake and Medallion architecture for scalable enterprise data platforms.
Develop ingestion frameworks for data from SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources.
Automate workflows using Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines.
Optimize Spark jobs for performance, reliability, and cost efficiency.
Implement data validation, quality checks, and monitoring with automated alerts and retries.
Design secure and governed datasets using Unity Catalog and cloud security best practices.
Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
Provide technical leadership and guidance to junior team members.

Required Skills

5–8+ years of experience in Data Engineering.
Strong hands-on experience with Databricks, PySpark, Delta Lake, SQL, Python.
Experience with Azure Data Lake, ADF, Azure Functions, or AWS equivalents (S3, Lambda).
Experience integrating data from APIs, SFTP servers, vendor data providers, and cloud storage.
Knowledge of ETL/ELT concepts, Lakehouse/Meddalion architecture, and distributed processing.
Strong experience with Git, Azure DevOps CI/CD, and YAML pipelines.
Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).

Good to Have

Exposure to Oil & Gas or trading analytics (SPARTA, KPLER, IIR, OPEC).
Knowledge of Power BI or data visualization concepts.
Familiarity with Terraform, Scala, or PostgreSQL.
Experience with SharePoint development or .NET (optional)

Apply

Apply at Unison Group

About the Company

Unison Group

Builds financial risk and compliance systems

Unison Group builds technology systems for financial institutions that need to manage risk, meet reporting rules and modernize their operations. Its consultants work across liquidity, market and credit risk, combining financial knowledge with software implementation and data analysis. The business also designs cloud infrastructure and develops custom applications, connecting these systems with the information and processes clients already use. It earns money through consulting, technology implementation and engineering services for enterprises, with work ranging from system design to deployment. Founded as Unison Consulting in 2012, the group is headquartered in Singapore.

Company size: 201-500
Company stage: Not listed
Total funding: Not listed
Headquarters: Not listed
Founded: 2012

Recently Posted Jobs

Explore more opportunities at Unison Group

See more jobs