Facebook pixel

Principal Data Engineer
Updated on 3/24/2023
Experience Level
Desired Skills
Apache Spark
Apache Kafka
Data Analysis
Natural Language Processing (NLP)
  • Scala, Java, Python
  • Apache Spark, Spark Streaming, Databricks/Delta Lake, Apache Storm, Elasticsearch, Apache Nifi
  • Kafka, MongoDB, Redis
  • AWS
  • Bachelor's degree (or higher) in Computer Science, Engineering, or similar and/or relevant work experience
  • Experience providing technical leadership at the enterprise level for the design of information technology systems
  • Crafted and implemented operational data stores, as well as data lakes in production environments
  • Ability to analyze, diagnose and resolve complex architectural problems using industry standard engineering principles
  • Design and build data ingestion pipelines and ETL processing, including stream processing, while factoring in performance and cost
  • Identify and solve issues concerning data management to improve data quality
  • Clean, prepare and optimize data for ingestion and consumption
  • Experience solving performance problems with Lucene based search solutions like Elasticsearch or Solr
  • 9+ years experience in server-side/back-end full cycle product development in a production environment
  • 4+ years developing with Apache Spark, including Structured Streaming. Experience with Databricks is a big plus
  • Knowledge of Scala or Java with exposure to or interest in Scala
  • Leads and mentors other team members
  • Provides partners with coaching and feedback in order to build effective teams
  • Provides effective support to cross-functional teams
  • Solve complex real-time data collection & analysis problems with cutting edge technical solutions
  • Iterate on our high performance and scalable platform for massive data collection, real-time analytics, NLP, machine learning, and backend data services
  • Build high performance, scalable, real-time, server-side technologies
  • Write scalable code with extensive test coverage, working in a professional software engineering environment with source control, dev/stage/production release cycles, continuous integration, and deployment
  • Work closely with product management, design, quality assurance and operations teams to understand our customers' needs and effectively translate them to technical specifications
  • Lead projects from translating product requirements into architecture to production
Zignal Labs

51-200 employees

Real-time intelligence data platform