Research Engineer

Job Description

At Canva, our mission is to empower the world to design. To ensure our generative AI models are truly helpful, we are seeking a talented Research/Machine Learning Engineer to build our next-generation evaluation system by leveraging automatic evaluations.

Job Description

At Canva, our mission is to empower the world to design. To ensure our generative AI models are truly helpful, we are seeking a talented Research/Machine Learning Engineer to build our next-generation evaluation system by leveraging automatic evaluations.

About the role:

You will engineer sophisticated AI agents that can automatically assess the quality and human alignment of our generative design models. This high-impact role focuses on building the practical systems that make cutting-edge research effective, to provide a rapid feedback loop that guides the future of design generation at Canva, ultimately empowering millions of users to create.

At the moment, this role is focused on:

• Agentic Evaluation Systems: Engineering autonomous AI agents that use Multimodal Large Language Models (MLLMs) to evaluate the quality, relevance, and human alignment of generated designs.

• Inference-Time Alignment: Mastering techniques that improve model outputs without full retraining, but by inference-based methods including prompt engineering, in-context learning and Retrieval-Augmented Generation (RAG).

• Model Benchmarking & Analysis: Building a rigorous framework to systematically benchmark internal and external quality understanding models, delivering clear, data-driven insights on human alignment.

Primary Responsibilities:

• Design, build, and optimize the infrastructure for an "MLLM-as-a-Judge" evaluation system for scalable, automated feedback.

• Implement and experiment with inference-time alignment techniques (Prompt Engineering, RAG, ICL) to directly improve model output quality.

• Establish and manage a comprehensive benchmarking process to compare various foundation models on design-centric tasks.

• Analyze evaluation data to identify model failure modes and provide actionable recommendations to the research team.

• Collaborate with research scientists and ML engineers to integrate the agentic judge system into the model development lifecycle.

• Translate the latest research in LLM evaluation and agentic AI into practical, production-ready engineering solutions.

You’re probably a match if you:

• You have a strong understanding of generative AI models (e.g., Diffusion Models, GANs, Transformers) and their architectures, with practical experience that informs robust evaluation strategies

• Excel at creating data-driven evaluation methodologies, turning user analytics into clear, actionable insights.

• You’ve successfully managed or optimized large-scale distributed model training across hundreds of GPUs

• You have a solid understanding of machine learning, have worked with PyTorch and know how to optimize such codes for speed

• You have disciplined coding practices, and are experienced with code reviews and pull requests.

• You have experience working in cloud environments, ideally AWS

Nice to Have:

• Familiarity with evaluation libraries and frameworks.

• Experience building or working with agentic AI systems or multi-agent coordination.

• Knowledge of data visualization tools to communicate findings effectively.

• A background or interest in human-computer interaction, design principles, or AI ethics.

Qualifications