Agentic Evaluation & Testing Specialist

Change the future with us.

We are looking for dedicated and talented people who solve ever-changing challenges, customer needs, and questions from colleagues with clever concepts and creativity. We embrace change and work with curious minds re-inventing the future of work. Join us and let us focus together on what’s truly important: making lives better with new ideas and the latest technology around the world.

About the role: System Testing and Agent Evaluation Specialist

We are seeking a System Testing and Agent Evaluation Specialist to join Siemens Digital Industries Operations Software at our AI hub in Canada. You make change verifiable in a product that is built AI-first: much of our code, tests, and documentation are produced by coding agents under the direction of engineers. This means that you will be working with the whole team on the challenge of validating the system quality.

You build the test infrastructure, the contract tests between components developed at our global sites, and the evaluation methods for agent behavior that cannot be judged by a simple pass or fail. You work with the industrial experts in Europe who define what a correct outcome looks like on a shop floor, and with the developer platform team that runs your tests at scale. This is a software engineering role for someone who thinks in terms of evidence and problem solving, not a manual testing role.

The role covers two separate workstreams with one underlying discipline. First, you are responsible for the verification of the product itself, covering classical software testing of a distributed, multi-site system. Second, you are responsible for building an evaluation workflow into the product that allows it to test and verify its own agent-generated output: new skills, prompts, and agent configurations are assessed for correctness, safety, performance, and cost in a controlled environment before a human decides whether to activate them. The first workstream is how we build the product; the second is a product feature our customers rely on.

About Siemens Digital Industries Operations Software: It is a suite of integrated digital solutions to engineer, simulate and operate manufacturing operations across industries. Operations Software includes our product portfolio in manufacturing engineering (COMOS, Tecnomatix, NX CAM, ...) & production operations (Win CC, Scada, PCS 7 / neo, Opcenter, Insights Hub, ...). The portfolio helps organizations across many different industries to improve operational efficiency, reduce costs, and enhance decision-making by providing real-time visibility and control across manufacturing and business processes.

Formula 1 cars, ships, mobile phones, space exploration vehicles, food & beverages, pharmaceuticals and many of the other objects we see in our daily lives are being conceived and produced using our software.

We're redefining the industrial world using data and AI to make production: Agile, Adaptive, Autonomous. Eliminate productivity boundaries for today’s and tomorrow’s production operation. Join our mission to build the most comprehensive next generation AI-native Industrial Operations Software portfolio as the new operation system for industry in the age of AI. With us you have the chance to now apply AI to where it really matters: In the physical world of an industrial production and engineering of real things.

As part of Siemens Operations Software (OPS-SW), the Technology, Data Fabric & Agentic AI organization provides the technology foundation that enables the next generation of industrial software. Its portfolio includes AI Production & Engineering Systems, Data Fabric, Agentic AI frameworks, connectivity platforms, semantic data models, cloud-native architectures, and industrial AI services that power autonomous, scalable, and intelligent operations across the OPS-SW portfolio.

What will you do?

• Build automated test infrastructure for a product developed across multiple sites, covering cross-component contract tests and integration and end-to-end tests against deployed systems.
• Design evaluation methods for agent behavior that a pass/fail assertion cannot capture: task completion, permission compliance, error handling, latency, and cost, measured over repeated runs and compared against the active version.
• Build the assessment workflow into the product itself, so that agent-generated skills, prompts, and configurations are evaluated in isolated environments before a human decides whether to activate them.
• Integrate the security checks defined by product security into that workflow, so that every artifact is assessed for correctness and containment in one run with two separate verdicts.
• Keep test expectations independent of the implementation, so that generated code and generated tests cannot confirm the same wrong assumption, and reject agent-generated tests that prove nothing.
• Make results usable by the product's own improvement loop as well as by human approvers, without letting anything bypass approval.
• Work AI-first: use coding agents for test development and failure analysis, and own the judgment about what the results mean.

What you will bring?

• Degree in computer science, software engineering, or a related field, or equivalent experience.
• 4+ years in software engineering with a focus on test automation, test infrastructure, or verification of distributed systems; you write production-quality code.
• Experience designing test strategies for systems with asynchronous behavior, external dependencies, and non-reproducible components.
• Practical experience with LLM-based applications and an understanding of why conventional assertions are not enough to evaluate them.
• Experience building evaluation pipelines or quality gates that run as part of a product or platform, not only in a developer's CI.
• Familiarity with contract testing, API schema validation, and CI/CD integration of test suites.
• Solid Python; comfortable with containerized test environments and Kubernetes-based execution.
• The instinct to ask "how would we know if this were wrong" before asking "does it pass".

What sets you apart?

• Experience with agent evaluation frameworks and AI observability tooling (Langfuse, LangSmith, Promptfoo, or comparable).
• Experience with property-based testing, metamorphic testing, or statistical test methods for nondeterministic systems.
• Background in safety-relevant or regulated software where test evidence is an audit artifact and release decisions are documented.
• Experience with simulation-based testing or hardware-in-the-loop setups.
• Exposure to industrial systems (MES, SCADA, OPC UA) and an understanding of what a wrong action costs on a shop floor.
• Experience designing systems that evaluate other AI systems, including LLM-as-judge setups and their known failure modes.

• Comfortable working with distributed teams across time zones and communicating asynchronously by default.

Salary is commensurate with experience, and ranges between $ 100,000 CAD - $ 150,000 CAD, excluding bonus and benefits. In addition to base salary, this role includes eligibility for an annual discretionary bonus of 10%.

Why you’ll love working for Siemens!

Join our Digital World

At Siemens Digital Industries Operations Software, flexibility is how we work – hybrid, built on trust and autonomy. We build technology that shapes the real world. You'll grow through real projects, strong technical peers, and global mobility, backed by the scale and benefits of an industrial operations software leader. We're committed to equality and inclusion, and we hire based on merit, skills, and impact. Bring your curiosity and creativity and help us shape tomorrow!

Our Benefits & Rewards

The actual compensation offered is based on the successful candidate's job-related skills, experience, and relevant education/training. Siemens offers a wide range of health and wellness benefits to employees

Our Commitment to Equity and Inclusion

We value equal opportunities and welcome applications from people with disabilities. At Siemens, we believe people who've had real experiences dealing with being different will excel as leaders. Let's foster a culture of creativity and innovation. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

Siemens is proud to be an eight-time award winner of Canada’s Top 100 Employers, Canada’s Greenest Employers 2025 and Canada’s Top Employers for Young People 2025.

About us.

We share our ideas and champion the people behind them.

Siemens Canada is a leading technology company focused on industry, infrastructure, mobility and healthcare. The company’s purpose is to create technology with purpose, transforming the everyday, for everyone, since 1912. By combining the real and the digital worlds, Siemens empowers its customers to accelerate their digital and sustainability transformations, making factories more efficient, cities more liveable, and transportation more sustainable. Siemens also owns a majority stake in the publicly listed company Siemens Healthineers, a leading global medical technology provider pioneering breakthroughs in healthcare. For everyone. Everywhere. Sustainably.

In fiscal 2024, which ended September 30, 2024, Siemens Canada had revenues of approx. $2.2 billion CAD. The company has approximately 4,400 employees from coast-to-coast and 37 office and production facilities across Canada.

To learn more about Siemens Canada, visit our website at www.siemens.ca

While we appreciate all applications we receive, we advise that only candidates under consideration will be contacted.

#OneOpsSW #LI-Hybrid