Sigma Software

Global software development and IT consulting

Senior Data Engineer

Data EngineeringConfirmed live in the last 24 hours

Full-timeWarsaw, PolandNo salary listedRemote

About the Job

Requirements

At least 5 years of experience in data engineering.
At least 3 years of production experience with Databricks and Delta Lake.
Strong expertise in medallion architecture and event-driven asynchronous pipeline design.
Advanced proficiency in Python, PySpark, and SQL.
Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization.
Strong AWS expertise, including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway.
Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles.
Practical experience with Git workflows, pull request reviews, and automated testing.
Experience configuring secure CI pipelines using OIDC or service principals.
Practical MLflow experience, including experiment tracking, model registration, and reproducibility practices.
Strong understanding of data quality, lineage, provenance, schema contracts, and versioning.
Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX.
Experience with monitoring, observability, and cloud cost optimization.
Upper-intermediate English level or higher.

Responsibilities

Design and build scalable cloud-native data platforms from greenfield to production.
Design and implement near-real-time ingestion pipelines using event-driven patterns.
Define and enforce data platform standards, including Data Lake/Lakehouse principles, medallion architecture, and data contracts.
Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability.
Introduce best practices for code quality, testing, and CI/CD across data pipelines.
Drive adoption of AI tooling and agentic workflows within the data engineering team.
Ensure data quality, observability, and reliability across pipelines and platforms.
Design and deliver self-service tooling and microservices that abstract platform complexity.
Collaborate directly with the customer's scientific and engineering teams.
Monitor platform performance, reliability, and cloud infrastructure costs.

Desired Qualifications

Experience packaging workloads in Docker containers and running batch jobs on AWS.
Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy.
Experience building feature sets and train/test splits that avoid data leakage.
Experience with regulated or high-security environments, including data residency requirements.
Experience disabling or restricting AI features in secured environments.

Job Description

Design and build scalable cloud-native data platforms from greenfield to production
Design and implement near-real-time ingestion pipelines using event-driven patterns
Define and enforce data platform standards including Data Lake/Lakehouse principles, medallion architecture, and data contracts
Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability
Introduce best practices for code quality, testing, and CI/CD across data pipelines
Drive adoption of AI tooling and agentic workflows within the data engineering team
Ensure data quality, observability, and reliability across pipelines and platforms
Design and deliver self-service tooling and microservices that abstract platform complexity
Collaborate directly with the Customer’s scientific and engineering teams
Monitor platform performance, reliability, and cloud infrastructure costs

Qualifications

5+ years of experience in data engineering
3+ years of production experience with Databricks and Delta Lake
Strong expertise in medallion architecture and event-driven asynchronous pipeline design
Advanced proficiency in Python, PySpark, and SQL
Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization
Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway
Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles
Practical experience with Git workflows, pull request reviews, and automated testing
Experience configuring secure CI pipelines using OIDC or service principals
Practical MLflow experience including experiment tracking, model registration, and reproducibility practices
Strong understanding of data quality, lineage, provenance, schema contracts, and versioning
Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX
Experience with monitoring, observability, and cloud cost optimization
Upper-Intermediate English level or higher

WILL BE A PLUS

Experience packaging workloads in Docker containers and running batch jobs on AWS
Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy
Experience building feature sets and train/test splits that avoid data leakage
Experience with regulated or high-security environments including data residency requirements
Experience disabling or restricting AI features in secured environments

Additional Information

PERSONAL PROFILE

Strong analytical mindset
Ownership and accountability
Ability to communicate complex technical concepts clearly
Proactive approach to problem-solving
Attention to scalability and maintainability
Collaborative and team-oriented attitude

About the Company

Sigma Software

Global software development and IT consulting

Sigma Software provides software development services and IT consulting to more than 170 clients worldwide. It builds large-scale software solutions and guides startups from concept to a ready product, across industries such as banking, automotive, advertising, telecommunications, cybersecurity, gaming, aviation, real estate, and healthcare. The company works by deploying teams of skilled engineers and consultants, delivering projects through global delivery centers and emphasizing clear communication, flexibility, and alignment with clients’ business goals. Sigma Software differentiates itself with recognition as a top outsourcing firm (IAOP World’s Top 100 Outsourcing), a wide geographic footprint, and the ability to take projects from idea through production for both established enterprises and startups. Its goal is to help clients achieve their business objectives by delivering reliable software products and expert IT advice on time and within scope.

Company size: 1,001-5,000
Company stage: Not listed
Total funding: Not listed
Headquarters: Stockholms kommun, Sweden
Founded: 2002

Recently Posted Jobs

Explore more opportunities at Sigma Software

See more jobs