M

Metropolitan Transportation Authority

Operates regional public transit network

Specialist Data Engineer

Data EngineeringConfirmed live in the last 24 hours
Full-timeNew York, NY, USA$114.1k - $134.6k

About the Job

Requirements

  • A Bachelor of Arts or Bachelor of Science degree is required; an equivalent combination of education and experience may be considered in lieu of a degree.
  • At least 3 years of relevant experience.
  • Strong knowledge of big data architectures, large data warehouses, and data lake solutions.
  • Experience designing and implementing modern data and analytics solutions, including lakehouse and medallion architectures.
  • Proficiency with cloud services including Azure Databricks, Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview, and Power BI.
  • Experience deploying and running cloud-based data solutions and infrastructure-as-code frameworks such as Terraform.
  • Proficiency in cloud deployments, preferably Microsoft Azure, in an agile software development lifecycle environment, leveraging modern programming languages and DevOps.
  • Experience with database platforms including Oracle, SQL Server, cloud databases, and NoSQL databases.
  • Strong understanding of data engineering concepts, ELT/ETL principles, and data modeling.
  • Experience with data integration techniques for structured and unstructured data.
  • Programming skills in Python, PySpark, and SQL.
  • Experience with Airflow.
  • Experience in DevOps, Git repositories, and CI/CD pipelines for code deployment.
  • Experience with programming and infrastructure-automation tools such as Terraform, Ansible, and Python.
  • Strong knowledge of Microsoft Azure Cloud.
  • Functional knowledge of Microsoft Power BI.
  • Experience with Jira and Confluence.
  • Ability to work independently and navigate organizational ambiguity.
  • Effective written and verbal communication skills.
  • Hands-on programming experience in a business setting.
  • Proficiency in at least one software engineering methodology, including Agile, Scrum, DevOps, Extreme Programming, Kanban, Lean, or Rapid Application Development.
  • Experience applying a structured validation and testing method, including unit, integration, system, acceptance, or regression testing.
  • Collaboration skills to ensure design specifications are implemented by the development team.

Responsibilities

  • Design, develop, and implement data solutions to meet business requirements and data ingestion needs, facilitating accurate and timely data availability for analysis and decision-making.
  • Extract, load, and transform data from various sources, including on-premises and cloud-based systems, APIs, databases, and files.
  • Write well-designed, efficient code that adheres to security standards.
  • Monitor and troubleshoot data pipelines to identify and resolve issues promptly and minimize disruptions in on-premises and cloud data processing.
  • Implement data quality checks and validation processes to ensure data accuracy and completeness, and collaborate with the data team to resolve issues.
  • Write complex queries and scripts to manipulate, transform, and process raw data efficiently.
  • Create and execute data validation processes to ensure incoming data reliability and consistency, and build processes to monitor data quality.
  • Continuously optimize data pipelines for performance, scalability, and reliability.
  • Create and maintain technical documentation and contribute to building and maintaining data catalogs and lineage.
  • Design and develop CI/CD processes that ensure high availability and agility.
  • Develop cloud data services provisioning automation with integrated IAM, network, security policies as code, and observability capabilities.
  • Build tools and services to support data discovery, lineage, resiliency, and privacy compliance across the data platform.
  • Stay updated on data engineering, cloud computing, and Azure services trends and best practices, and suggest solutions to improve the organization’s data intelligence capabilities.
  • Monitor and report on supplier performance, customer satisfaction, adherence to security requirements, and market intelligence.
  • Mentor less experienced staff as needed.

Desired Qualifications

  • Microsoft Certified: Azure Data Engineer Associate certification.
  • Microsoft Certified: Azure Data Fundamentals certification.
  • Experience with Microsoft Purview.
  • Knowledge of AWS and GCP.
  • Thorough knowledge or full proficiency in data structures and algorithms.
  • Thorough knowledge or full proficiency in cloud computing.

About the Company

Metropolitan Transportation Authority

Operates regional public transit network

The Metropolitan Transportation Authority operates the New York City region’s transit network, including buses, subways, commuter rails, bridges, and tunnels, through six agencies. It moves people by running trains and buses and by maintaining toll facilities and related infrastructure to deliver daily service. It stands apart from others by being the largest multi-agency transit authority in the U.S., offering integrated services across multiple modes under one umbrella. Its goal is to provide safe, reliable, and accessible transportation that keeps the New York City area moving for millions of riders.

Company size
1,001-5,000
Company stage
Not listed
Total funding
Not listed
Headquarters
New York City, New York
Founded
1834

Recently Posted Jobs

Explore more opportunities at Metropolitan Transportation Authority

See more jobs