Senior HPC Storage Architect

The Research & Development team at Hudson River Trading (HRT) builds and maintains the computers, networks, storage, operating systems, and software that allow our trading strategies and research environment to operate worldwide 24/7. HRT operates incredibly large GPU and CPU compute clusters, larger than most national labs. We have built or are in the process of building our own on-prem data centers globally and manage over 100 petabytes of storage to facilitate industry leading AI/ML based trading.

We are looking for a hands-on Storage Architect who enjoys being challenged, appreciates an open and collaborative organizational structure, and thrives in a fast-paced environment. In this role, you will own the vision, planning, and execution of HRT’s current and future storage needs. Given HRT’s growth and ever-evolving needs (especially when it comes to AI), this is a unique opportunity to set the direction for future storage solutions and work across engineering teams to deliver it. You will drive engagement with all stakeholders — internal users and both external and internal technology providers — understand their product portfolio and roadmap, and build a strategy for HRT that is inclusive of financial modeling.

Responsibilities

• Own the overall charter of delivering performant, reliable, and future-proof storage solutions for all of HRT’s storage needs
• Collaborate and drive efforts across multiple (internal) cross-functional engineering teams
• Improve user experience by understanding storage workloads and designing a variety of solutions and tools
• Forecast future storage needs, research new storage solutions, and build a five- and ten-year roadmap
• Draw insights from a deep understanding of hardware infrastructure and industry trends to inform the roadmap and planning decisions
• Review deliverables and provide relevant guidance to engineering teams
• Troubleshoot complex storage, OS, and networking issues
• Integrate storage systems with HPC clusters, ensuring compatibility with existing hardware and software
• Participate in an on-call rotation to ensure continuous support for HRT’s operations, responding promptly to critical storage issues and incidents.

Qualifications

• 10+ years of experience in HPC-like environments with multi-PB storage deployments
• Experience in developing advanced storage software solutions and/or leading software development team(s)
• Deep expertise in HPC and scale-out enterprise storage solutions
• Knowledge of distributed file systems used for large-scale cluster computing (Lustre, GPFS, WEKA, S3, CEPH, etc.)
• Strong leadership, communication, and stakeholder management skills
• Deep technical understanding of commodity storage technologies
• Knowledgeable on storage industry trends
• Deep technical understanding of server architecture, design, and development, with emphasis on power and performance analysis
• Ability to analyze and solve problems under quick turnaround times
• Ability to manage time efficiently, balancing independent and collaborative workflows
• Proficient in Python and UNIX/Linux shell scripting