Facebook pixel

Lead Data Engineer
Confirmed live in the last 24 hours
Experience Level
Desired Skills
Apache Spark
Apache Kafka
Data Analysis
  • Advanced knowledge of data architectures, data pipelines, real time processing, streaming, networking, and security
  • Hands on experience programming in Python, Scala or Java
  • Expertise with RDBMS and Data Warehousing (Strong SQL) with Redshift,Snowflake or similar
  • In-depth knowledge and experience with data and information architecture patterns and implementation approaches for Operational Data Stores, Data Warehouses, Data Marts and Data Lakes
  • Proficiency in logical/physical data architecture, design and development
  • Experience in Data lake / Big data analytics platform implementation either cloud based or on-premise; AWS preferred
  • Experience working with high volumes of data; experience in design, implementation and support of highly distributed data applications
  • Experience with Development Tools for CI/CD, Unit and Integration testing, Automation and Orchestration E.g. GitHub, Jenkins, Concourse, Airflow, Terraform
  • Experience with writing Kafka producers and consumers or experience with AWS Kinesis
  • Hands-on experience developing a distributed data processing platform with Big Data technologies like Hadoop, Spark etc
  • A knack for independence (hands-on) as well as team work
  • Excellent analytical and problem-solving skills, often in light of ill-defined issues or conflicting information
  • Experience with streaming data ingestion, machine-learning, Apache Spark a plus
  • Adept in the ability to elicit, gather, and manage requirements in an Agile delivery environment
  • Excellent communication and presentation skills (verbal, written, presentation) across all levels of the organization. Ability to translate ambiguous concepts into tangible ideas
  • Lead multiple projects and efforts to orchestrate and deliver cohesive data engineering solutions in partnership with various functional teams at Thrive Market
  • Be hands-on and take ownership of the complete cycle of data services, from data ingestion, data processing, ETL to data delivery for reporting
  • Collaborate with other technical teams to deliver data solutions which meet business and technical requirements; define technical requirements and implementation details for the underlying data lake, data warehouse and data marts
  • Identify, troubleshoot and resolve production data integrity and performance issues
  • Collaborate with all areas of data management as lead to ensure patterns, decisions, and tooling is implemented in accordance with enterprise standards
  • Perform data source gap analysis and create data source/target catalogs and mappings
  • Develop a thorough knowledge and understanding of cross system integration, interactions and relationships in order to develop an enterprise view of Thrive Market's future data needs
  • Design, coordinate and execute pilots/prototypes/POC to provide validation on specific scenarios and provide implementation roadmap
  • Recommend/Ensure technical functionality (e.g. scalability, security, performance, data recovery, reliability, etc.) for Data Engineering
  • Facilitate workshops to define requirements and develop data solution designs
  • Apply enterprise and solution architecture decisions to data architecture frameworks and data models
  • Maintain a repository of all data architecture artifacts and procedures
  • Collaborate with IT teams, software providers and business owners to predict and devise data architecture that addresses business needs for collection, aggregation and interaction with multiple data streams
  • Accountable for the delivery of short to mid term goals for assigned pods/initiatives
  • Responsible for the team's overall output, and meeting deadlines on projects
  • Assumes project management responsibilities for core work or key initiatives
  • Contributes significantly to the development of features with minimal guidance
  • Complete subtasks daily
  • Fixes defects permanently within a reasonable timeframeWrites thorough documentation, comments, and automated tests
  • Demonstrates full knowledge of data tool features, configurations, and systems in functional areas
  • Develops new ideas for teams to produce results more efficiently and rapidly
  • Mentors mid-level and senior engineers to continuously improve performance
  • Collaborates across pods, functional teams, and departments to ensure projects and features are managed to minimize roadblocks
  • Communicates clearly with team members and departments, and delivers meaningful updates to Directors and other Management
  • Demonstrated growth mindset by increasing the quality and velocity of team's work as well as their own leadership capabilities
  • Can manage projects successfully along with timelines and provides technical leadership to the team
Thrive Market

501-1,000 employees

Company Overview
Thrive Market is an online retailer with over 2,500 high-quality healthy foods, vitamins, home goods, and beauty supplies.