This team will provide 24x7 white-glove support to people using large blocks of GPUs (6,000+ contiguous GPUs) for a short period of time (eg: 6-weeks, 12-weeks etc) to perform Managed Post Training (MPT). This includes helping with preparation, 24x7 support during training to ensure full utilization of the GPU clusters and off-boarding. The team is in three timezones with hand-off protocols to enable 24x7 support: US, Europe and India. While you can be a specialist in Infra and cluster operations, you need to know enough about ML.

• You will help shape and iterate this new white glove model training support service on large GPU clusters.
• You will work in a collaborative team with Machine Learning Engineers and Infrastructure Engineers.
• You will contribute to accelerator development: find gaps in the tooling, or needed automation, or patterns we would develop accelerators to make the next round of this more efficient and faster. Eg: We need to improve observability, or we need to automate user onboarding, or we need to bring in a new tool which everyone seems to want to use etc. This will probably involve a combination of Terraform/Pulumi, Helm Charts, Python and Shell Scripts.
• You will help assess the model training readiness and data preparation.
• You will provide model training support rotating daytime weekend shifts - with pagers, to any issues they may encounter. These can range from infrastructure issues to data sciences issues or anything in between: eg: GCP changed a configuration in GKE that affects the training.
• You will facilitate collaborative problem solving within the team by actively listening, communicating effectively and mentoring other engineers.
• You will proactively identify and address challenges related to the white glove service for continued pre training, proposing solutions and implementing improvements.

Technical Skills

Professional Skills

Learning & Development

Thoughtworks is a dynamic and inclusive community of bright and supportive colleagues who are revolutionizing tech. As a leading technology consultancy, we’re pushing boundaries through our purposeful and impactful work. For 30+ years, we’ve delivered extraordinary impact together with our clients by helping them solve complex business problems with technology as the differentiator. Bring your brilliant expertise and commitment for continuous learning to Thoughtworks. Together, let’s be extraordinary.