Principal Data Engineering Lead
Posted on 11/30/2023
INACTIVE
48forty Solutions

201-500 employees

Comprehensive pallet management services across North America
Company Overview
48forty Solutions stands as the largest pallet management services company in North America, boasting a vast network of over 270 facilities and a fleet that is entirely company-owned, ensuring consistent quality, pricing, and a higher level of service. The company's comprehensive services, ranging from nationwide pallet supply and retrieval to on-site services and reverse logistics, make it a one-stop solution for over 2,700 customers across diverse industries. With local teams that understand the nuances of each market, 48forty Solutions offers a unique blend of industry leadership, technical prowess, and a customer-centric culture.
Industrial & Manufacturing

Company Stage

Grant

Total Funding

$25K

Founded

1992

Headquarters

Houston, Texas

Growth & Insights
Headcount

6 month growth

0%

1 year growth

-2%

2 year growth

34%
Locations
Houston, TX, USA
Experience Level
Entry
Junior
Mid
Senior
Expert
Desired Skills
Power BI
Microsoft Azure
Agile
Data Structures & Algorithms
Apache Spark
SQL
Tableau
AWS
Data Analysis
CategoriesNew
Data & Analytics
Requirements
  • 5 years of experience in data engineering, data integration, and data warehousing using Azure services such as SQL Server, Azure Data Factory, Azure Synapse, and Azure Databricks.
  • 5 years of experience in SQL programming and expertise in working with database management systems like SQL Server.
  • Bachelor's Degree in Computer Science, Information Technology, or a related field required.
  • In-depth understanding of dimensional modeling principles and best practices for designing scalable and high-performance data structures.
  • Hands-on experience with Spark SQL and data processing using Apache Spark.
  • Familiarity with cloud-based architecture and services, especially Microsoft Azure.
  • Knowledge of data governance, data quality, and data security principles.
  • Experience working in an Agile development environment and familiarity with Agile methodologies.
  • Strong understanding of Azure DevOps practices and tools for version control, CI/CD, and automation.
  • Knowledge and experience with DataOps principles, including data pipeline testing, deployment automation, and monitoring.
  • Strong problem-solving and Analytical Acumen
  • Strong knowledge of cloud-based architectures and services, particularly Microsoft Azure.
  • Proficiency in data governance, data quality, and data security principles.
  • Experience leading and mentoring a team of data engineers, providing technical guidance, and driving best practices.
  • Expertise in Agile methodologies and experience working in Agile development environments.
  • Ability to multi-task, prioritize assignments and work well under deadlines in a changing environment with a cross functional agile team.
  • Experience managing and leading a small team is preferable.
Responsibilities
  • Lead a team of data engineers, providing technical guidance, mentoring, and coaching. Drive the adoption and establishment of recommended practices, standards, and methodologies for data engineering projects.
  • Collaborate with stakeholders to understand data requirements and translate them into scalable and performant data engineering and data product solutions. Design data pipelines, data models, and system architectures using Azure services, including Azure Data Factory, Azure Synapse, and Azure Databricks.
  • Lead the development of data pipelines, ELT processes, and data integration workflows using Azure Databricks, Azure Synapse Pipelines, Spark SQL, PySpark and other relevant technologies. Ensure the availability, reliability, and scalability of data processing systems.
  • Apply dimensional modeling techniques to design and implement data models in SQL Server, adhering to best practices for analytical reporting and business intelligence. Design star schemas, fact tables, and dimension tables to support data analysis and reporting requirements.
  • Optimize data processing workflows, SQL queries, and transformations to improve system performance, reduce latency, and enhance scalability.
  • Establish data quality standards, implement data validation rules, and develop data cleansing processes. Ensure adherence to data governance policies and practices to maintain the accuracy, integrity, and security of data.
  • Collaborate in an Agile team, participate in sprint planning, backlog refinement, and other Agile ceremonies.
  • Implement and oversee Azure DevOps practices for version control, CI/CD, and automation of data engineering artifacts. Drive the adoption of DataOps principles, including testing, deployment automation, and monitoring of data pipelines.
  • Collaborate effectively with stakeholders, including data analysts, and business users, to understand their data requirements and align data engineering solutions accordingly. Communicate technical concepts and project progress to non-technical stakeholders.
  • Stay up to date with the latest advancements in data engineering technologies and tools. Maintain and achieve relevant certifications. Evaluate, establish, and recommend methodologies, and frameworks that can enhance the data engineering capabilities of the team.
Desired Qualifications
  • Experience managing and leading a small team
  • Certifications in relevant data engineering technologies
  • Experience with other cloud platforms such as AWS or Google Cloud
  • Experience with data visualization tools such as Power BI or Tableau