Principal Data Lead - Data Engineering with Full-Stack Dev, Investment Systems

POSITION SUMMARY
We are seeking a Principal Data Lead to set the technical direction of our Reference Data Platform — the system of record that masters, governs, and distributes the firm’s core reference data (parties, instruments, portfolios, and accounts. This is a hands-on technical leadership role: you will own the end-to-end architecture of a layered, event-driven data platform — spanning sourcing, ingestion, entity mastering, a bitemporal golden copy, and an API-first distribution layer — and drive it from vision to production. You will be the senior-most engineering voice on reference data, shaping platform strategy, establishing engineering standards across teams, and partnering with product, data, and infrastructure leaders to deliver a scalable, vendor-agnostic master data capability. The ideal candidate is an expert Python engineer and data architect with deep experience in master data management (MDM) and reference data within the financial industry — including market data and vendor feeds — combined with hands-on experience applying Generative AI to data platforms, distributed data processing, and building reusable, independently scalable platform components.

We are operating in a 4-day in office, 1-day flexible work arrangement. 

ROLES & RESPONSIBILITIES
•    Own the architecture of a layered reference-data platform, defining the contracts between sourcing, ingestion, mastering, golden-copy, and distribution layers so each is independently buildable, deployable, and scalable.
•    Set technical direction for entity mastering — standardization, blocking, matching, survivorship, and crosswalk/lineage — designing rules as configuration (data, not code) that scale from exact-match fast paths to full-population fuzzy resolution.
•    Architect the durable golden copy: a bitemporal, point-in-time source of truth with match evidence, per-attribute provenance, and a schema-last serving model that adapts to new sources and domains without migration.
•    Define the event-driven backbone and cadence model (start-of-day, intraday, end-of-day, and on-demand) enabling batch today and real-time streaming (CDC / Kafka) without re-architecture.
•    Design the API-first, self-documenting distribution surface (REST, curated data products, subscriptions) governed by data contracts and consumable by applications, analytics, and AI/agents.
•    Drive the platform’s storage and compute strategy over an open lakehouse (Apache Iceberg, Snowflake) and elastic compute (Spark), ensuring an open, vendor-agnostic, no-lock-in foundation.
•    Establish engineering standards, reference patterns, and a reusable component library (a layered monorepo) that make capabilities build-once, reuse-everywhere across domains.
•    Lead cross-team technical design reviews, resolve the hardest architectural trade-offs, and align engineering execution with the firm’s data strategy.
•    Champion data quality, governance, security, entitlements, audit, and observability as first-class, cross-cutting foundations of the platform.
•    Mentor senior and staff engineers, grow technical depth across the team, and set the bar for engineering excellence.
•    Partner with product and business stakeholders to translate reference-data needs into a durable, scalable platform roadmap.

QUALIFICATIONS

•    Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
•    12+ years of professional software engineering experience, including significant time as a technical lead or architect of data-intensive platforms.
•    Expert-level proficiency in Python for building production services, data pipelines, and platform frameworks.
•    Proven experience architecting master data management (MDM) or reference/entity-data systems — entity resolution, survivorship, golden records, crosswalk, and lineage.
•    Deep data engineering expertise: data modeling, ETL/ELT design, distributed processing (Spark), and data distribution patterns at scale.
•    Hands-on experience designing API-first services and event-driven / streaming architectures (e.g., Kafka, CDC).
•    Strong command of modern data storage — relational, non-relational, and lakehouse formats (e.g., Apache Iceberg) — and SQL.
•    Experience with infrastructure-as-code (Terraform) and cloud-native deployment (containers, Kubernetes/EKS, CI/CD).
•    Deep domain experience with reference and master data in the financial industry — parties / legal entities, instruments, portfolios, and accounts — and the identifier, hierarchy, and cross-referencing challenges specific to it.
•    Hands-on experience integrating and mastering market data and vendor/administrator feeds (e.g., Bloomberg, exchange, and fund-servicing sources), including normalization, cross-referencing, and point-in-time handling.
•    Practical experience applying Generative AI / LLMs to data platforms — e.g., grounded retrieval over governed data, entity-resolution and data-quality assistance, or AI/agent-ready data access with provenance guarantees.
•    A track record of setting technical direction, driving architecture across teams, and mentoring senior engineers.
•    Excellent judgment on architectural trade-offs, and the communication skills to align engineering, product, and business stakeholders.

PREFERRED QUALIFICATIONS
•    Familiarity with bitemporal / point-in-time data modeling and governed, auditable data systems.
•    Experience with workflow orchestration (e.g., Dagster, Airflow) and lakehouse catalogs (e.g., Polaris, Nessie, Glue).
•    Exposure to commercial or vendor MDM/ontology platforms (e.g., Palantir Foundry) and experience integrating them as pluggable engines within an open architecture.
•    Experience building agentic workflows or retrieval-augmented systems on top of enterprise data, and familiarity with LLM evaluation, guardrails, and grounding techniques.
•    Experience with legal-entity identifiers and standards (e.g., LEI, GLEIF) and regulatory reference-data requirements.