Data Engineer - Unified Data Layer

Job Description

Platform & Pipeline Engineering

• Develop, automate, and maintain resilient data pipelines connecting on-premises 
• laboratory equipment, high throughput test benches, and edge systems to Microsoft Azure.
• Investigate, unpack, and parse complex, non-standard, binary, proprietary, and  undocumented file formats where standard connectors do not exist.
• Implement parsers and normalization routines capable of ingesting diverse technical data streams, for example:

1. High-frequency sensor streams, time series, and bus telemetry.
2. Multi-dimensional scientific arrays, spatial grids, and spectral data.
3. - Hierarchical, nested, and unstructured binary logs or custom ASCII exports.
4. Event logs, operational metadata, and intermittent experimental batch runs.

• Clean, validate, and transform fragmented or schema-less raw data into optimized, partitioned Lakehouse tables (Delta Lake/Parquet) ready for analytics and ML model training.
• Deploy, monitor, and maintain pipeline resources on Azure using Terraform, Databricks, and containerized workflows following established team templates.

Collaboration & Lab Support

• Work directly alongside experimental scientists, lab engineers, and technicians to decipher raw measurement files, clarify domain-specific semantics, and automate recurring data uploads.
• Collaborate closely with solution architects and senior engineers to implement data ingestion patterns, cataloguing schemas, and governance standards.
• Maintain clear technical documentation, code comments, and onboarding walkthroughs to help research teams connect their equipment seamlessly.
• Support proof-of-concept initiatives exploring metadata integration with corporate AI assistants, data discovery catalogs, and modern data tools.

Qualifications

Education

• BSc, MSc or Ph.D. in Data Engineering, Computer Science, Software Engineering, Applied Physics, Electrical Engineering, or a related field.

Experience

• 3+ years of professional experience in data engineering, building automated and reliable data pipelines.
• 1–2 years of practical experience handling heterogeneous, non-standard, or binary data sources (e.g., physical sensor logs, laboratory exports, hardware telemetry, or raw byte streams).
• 1+ years of hands-on experience utilizing Apache Spark (preferably Databricks) and Microsoft Azure in production environments.
• Demonstrated experience deciphering, parsing, or converting non-standard, legacy, or custom data formats into structured data structures.

Technical Skills

• Strong proficiency in Python for data manipulation (including binary data handling using modules like struct, byte manipulation, NumPy, Pandas, and PySpark) and Bash.
• Solid understanding of modern Lakehouse concepts (Delta Lake), distributed data processing, SQL, and NoSQL/document datastores.
• Proficient with Git, GNU/Linux environments, and working knowledge of Infrastructure as Code (Terraform).
• Practical experience with standard transport mechanisms (e.g., REST APIs, SFTP, Kafka/Event Hubs, or IoT/streaming protocols).

Mindset & Soft Skills

• Strong analytical mindset with a natural curiosity to open unfamiliar files, inspect headers, and solve data decoding puzzles.
• Friendly, empathetic communication style with the ability to bridge conversations between laboratory technicians, domain scientists, and software teams.
• C2 CEFR level in English, written and spoken.

Nice to Have

• Experience with scientific/hierarchical file formats (e.g., HDF5, NetCDF, Parquet, Protobuf).
• Familiarity with industrial telemetry or time-series historian protocols (e.g., OPC UA, MQTT).
• Exposure to Azure analytics tooling (Azure Data Factory, Azure Event Hubs, Azure Data Explorer).
• Experience with vectorial databases.

Languages

• Fluent in Portuguese.
• B2 CEFR level in English, written and spoken.
• B2 CEFR level in German, written and spoken is a plus

Additional Information

What we offer you:

Challenges in an international environment | career opportunities in one of the best companies in the world | flexible hours | medical services | discounts for employees | sports and health related activities | canteen | good access to public transports | space for creativity.

Success stories don´t just happen. They are made...

Make it happen! We are looking forward to your application!