P

Pfizer

Makes cancer drugs, vaccines and other medicines

Manager, AI Ops Site Reliability Engineer

AI Application EngineeringConfirmed live in the last 24 hours
Full-time€31.1k /yr

About the Job

Requirements

  • A bachelor's degree in a technical field or equivalent practical experience.
  • At least 4 years of experience in software, data, or machine learning/artificial intelligence engineering, including hands-on production delivery of large language model or agent-based applications.
  • Demonstrated prompt and context engineering, and experience building structured evaluations for AI systems, including ground-truth sets, metrics, and regression testing; large language model-as-judge experience is a plus.
  • Working knowledge of agent frameworks and tooling, such as Amazon Bedrock AgentCore or LangGraph, retrieval-augmented generation and knowledge-base design, and strong Python skills.
  • Information technology operations literacy to design realistic operational workflows and judge the plausibility of agent output; domain administrator experience is not required.
  • Strong analytical and communication skills to work with site reliability engineers, extract knowledge, and present accuracy results credibly.
  • Demonstrated experience in an agile work environment.
  • Periodic international and domestic travel of less than 10% is required.
  • Applicants must submit their CV in English.

Responsibilities

  • Design agent workflows for intake and investigation, including prompts, context and retrieval, tool selection, and decision logic that determine how the agent classifies, enriches, and reasons about incidents.
  • Build and own the evaluation harness, including ground-truth datasets drawn from historical incidents, accuracy, precision and recall metrics, regression tests, and hallucination and error detection, so each agent change is measured before release.
  • Define reasoning guardrails for what an agent may conclude and propose, the required confidence level, and when it must defer to a human.
  • Work with domain experts to convert runbooks and past investigations into retrievable knowledge that agents can cite.
  • Redesign operational workflows with domain owners as agent-assisted flows, pilot them, and measure reductions in toil and mean time to recovery.
  • Continuously tune agent quality against production results and feed misses back into prompts, context, and evaluation sets.
  • Collaborate with site reliability engineering, operations, platform teams, AI Ops platform engineers, AI Ops site reliability engineers, and other stakeholders to align operational requirements, platform capabilities, automation strategies, and execution priorities.

Desired Qualifications

  • Experience using large language model-as-judge evaluation.
  • Familiarity with ServiceNow, Dynatrace, or observability data.

About the Company

Pfizer

Makes cancer drugs, vaccines and other medicines

Pfizer discovers, makes, and sells prescription medicines and vaccines for patients, doctors, and governments worldwide, with large businesses in vaccines, cancer, heart disease, and rare diseases. In 2020, its COVID-19 shot, developed with German partner BioNTech, became the first COVID-19 vaccine authorized in the United States. Its scientists find drug candidates and run clinical trials to prove they are safe and effective, and Pfizer then manufactures approved products in its own plants, earning most of its money from patented drugs before cheaper generic copies arrive. Pfizer was founded in 1849 as a chemicals business in Brooklyn and is based in New York City.

Company size
10,001+
Company stage
IPO
Headquarters
New York City, New York
Founded
1849

Recently Posted Jobs

Explore more opportunities at Pfizer

See more jobs