AI Engineer - Evaluation

AI Singapore (AISG) is a national initiative driven by the Ministry of Digital Development and Information (MDDI), the Infocomm Media Development Authority (IMDA), and the National Research Foundation, Singapore (NRF). 

AISG brings together Singapore-based research institutions and the broader ecosystem of AI start-ups and companies to support impactful research, develop talent and power Singapore's AI efforts. 

For more information on AI Singapore, please visit https://www.aisingapore.org.

This position will be hosted at the Nanyang Technological University (NTU) under VP (Artificial Intelligence & Digital Economy)’s office and we welcome you to join our community.

Duties and Responsibilities:

• Develop and maintain evaluation frameworks and pipelines to measure the capabilities of Large Language Models (LLMs).
• Keep up to date and experiment with the latest research in multilingual, multicultural, and multimodal LLM evaluations such as the LLM-as-a-Judge paradigm.
• Work with partners to collect, translate and verify evaluation datasets.
• Perform the necessary data preparation and analysis, AI modelling, coding, testing, validation and deployment to ensure reliable and scalable AI solutions.
• Collaborate with cross-functional teams within AI Products to design and resolve issues.
• Maintain code repository and documentation standards.
• Contribute to community engagement activities such as sharings via technical session meet-ups and article write-ups, and participating in discussion forums.

Requirements:

• Graduate from a degree program in computer science, AI, data science or related fields (or equivalent practical experience).
• Deep understanding of LLM evaluation experimental design and the advantages/limitations of various LLM evaluation methods.
• Experience in LLM inference frameworks such as vLLM and AI/Deep Learning frameworks such as PyTorch.
• Writing production level code in Python and using version control systems such as Git.
• Strong written and verbal communication skills.
• Independent learner that is capable of reading and understanding research papers.
• Fluent in English and one other Southeast Asian language for the purposes of understanding how to build quality evaluations from a multilingual and multicultural perspective.

We regret that only shortlisted candidates will be notified.

Hiring Institution: NTU