AI Operations Analyst

At Johnson & Johnson, we believe health is everything. Our strength in healthcare innovation empowers us to build a world where complex diseases are prevented, treated, and cured, where treatments are smarter and less invasive, and solutions are personal. Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity. Learn more at jnj.com.

As guided by Our Credo, Johnson & Johnson is responsible to our employees who work with us throughout the world. We provide an inclusive work environment where each person is considered as an individual. At Johnson & Johnson, we respect the diversity and dignity of our employees and recognize their merit.

Job Function:

Technology Product & Platform Management

Job Sub Function:

Technical Product Management

Job Category:

Scientific/Technology

All Job Posting Locations:

Lisbon, Portugal, Madrid - Lafinca, Milano, Italy

Job Description:

Role Overview

The AI Operations Analyst manages day-to-day operational analysis and monitoring for AI applications and AI agents, helping maintain reliable, secure, transparent, and cost-effective services. The role combines hands-on investigation, dashboard and reporting ownership, incident coordination, and continuous improvement of established operational processes.

The analyst takes ownership of assigned operational activities across observability, consumption reporting, incident triage, audit evidence, and service readiness. The role works with AI Operations engineers and partner teams to investigate issues, recommend improvements, and implement approved changes within defined access and change-management controls. The analyst also supports solution design by contributing operational insights and practical requirements for monitoring, supportability, and cost visibility.

The role does not own the end-to-end lifecycle management of agents or AI products, platform architecture, or final security and compliance approvals. Agent design, development, functional evaluation, release content, product evolution, and retirement decisions remain with the Agent Factory and the relevant AI product teams. The analyst independently delivers routine operational work and coordinates more complex investigations with engineers and accountable control owners.

Key Responsibilities

AI Observability & Production Monitoring

·       Own routine monitoring of AI applications and agents, analysing logs, traces, model and tool calls, retrieval steps, latency, failures, and token consumption to identify service issues.

·       Validate telemetry completeness across agent workflows and supporting services, investigate gaps, and coordinate corrective actions with engineers and platform owners.

·       Build and maintain operational dashboards and reports covering availability, latency, errors, quality signals, policy alerts, consumption, and service health.

·       Analyse performance against agreed service-level indicators and objectives, tune operational alert thresholds within approved standards, and recommend improvements to monitoring coverage.

·       Investigate incidents using logs and traces, develop evidence-based hypotheses, and coordinate next steps while following approved access and sensitive-data handling procedures.

·       Investigate agent loops, retrieval issues, degraded tool performance, and unexpected runtime behaviour; prioritise findings and escalate complex defects to engineering teams.

·       Own incident analysis and follow-up for assigned operational issues, contribute to root-cause reviews, and implement agreed preventive actions within the role’s technical authority.

LLMOps & AgentOps Support

·       Maintain configuration, prompt, model, and agent-component version records, and improve the consistency of operational checks across environments.

·       Execute and improve deployment-readiness checks, investigate failed integration, regression, security, or observability checks, and coordinate resolution with responsible teams.

·       Coordinate operational checks and monitoring during rollouts, validate fallback and rollback readiness, and execute authorised operational actions under approved change procedures; release approval remains with accountable owners.

·       Maintain consistent use of shared operational tools and identify opportunities to standardise support processes across models, frameworks, and agent patterns. Support engineers and product teams during design by helping define telemetry, alerting, troubleshooting, and recovery requirements; design decisions remain with accountable engineering and product owners.

·       Analyse functional evaluation results supplied by the Agent Factory or AI product teams alongside operational signals, flag regressions, and support readiness reviews; functional quality ownership remains with those teams.

·       Own the accuracy and maintenance of assigned runbooks, checklists, knowledge articles, and reusable operational examples.

AI Consumption & Cost Reporting

·       Own recurring consumption and cost reporting by product, agent, workflow, model, environment, or business unit, validating allocation data and explaining material variances.

·       Analyse token consumption, model usage, repeated loops, retrieval overhead, infrastructure usage, and cost-per-transaction metrics to identify inefficiencies and emerging trends.

·       Investigate consumption anomalies and budget alerts, recommend threshold adjustments, and coordinate approved containment actions with engineers and budget owners.

·       Recommend optimisation opportunities in model selection, context size, caching, retries, and tool usage, partnering with engineers and product teams to assess quality and compliance impacts.

·       Track the outcomes of approved optimisation actions and provide cost and usage analysis for capacity planning; budget and investment decisions remain with accountable leads.

Security & Compliance Operations Support

·       Apply approved security, privacy, Responsible AI, and compliance requirements to operational activities, identifying control gaps and coordinating remediation with the relevant owners.

·       Coordinate routine access reviews, investigate discrepancies, and track corrective actions; permission changes and secrets administration remain with authorised owners.

·       Triage and investigate runtime security alerts, including suspected prompt injection, unauthorised tool use, data leakage, and anomalous access; coordinate response with security and engineering teams.

·       Validate that operational logging, retention, redaction, and access practices follow approved requirements, and resolve or escalate identified deviations.

·       Maintain complete and accurate operational evidence linking versions, configurations, identities, actions, approvals, and outcomes.

·       Coordinate recurring policy checks and evidence collection, analyse gaps, and track remediation to closure with accountable control owners.

·       Contribute to security testing and incident response through operational investigation, evidence analysis, and delivery of assigned remediation actions.

Operational Service Support & Documentation

·       Manage assigned monitoring, support, incident, problem, and change activities in accordance with agreed service processes and escalation paths.

·       Coordinate operational-readiness assessments, validate service acceptance evidence, and maintain runbooks, escalation paths, and recovery documentation.

·       Coordinate operational handoffs across AI Operations, the Agent Factory, product teams, and platform owners, ensuring open issues have clear owners and follow-up actions.

·       Develop and maintain scripts or workflow improvements that automate repetitive reporting, validation, and support tasks, using approved testing and change controls.

·       Maintain operational knowledge articles and share practical guidance to improve consistency and effectiveness across support teams.

Stakeholder Collaboration

·       Act as the operational point of contact for assigned services, coordinating updates, investigations, and information requests across AI Operations and partner teams.

·       Translate operational findings into clear requirements, prioritised issues, and improvement recommendations for engineering and control owners; contribute to design reviews by highlighting operational lessons, support needs, and potential service risks.

·       Promote adoption of approved operational practices and resolve routine process questions, escalating architectural or policy decisions to accountable owners.

Governance, Risk & Compliance

·       Coordinate assigned governance and compliance activities, validate evidence completeness, and raise unresolved risks or exceptions to accountable owners.

·       Maintain traceability of controls, exceptions, evidence, ownership, and remediation actions, following up on overdue items and reporting status.

Metrics & Continuous Improvement

·       Own recurring KPI reporting for assigned services, including reliability, observability coverage, incident response, consumption, control coverage, audit evidence, and adoption.

·       Analyse operational trends, identify recurring issues, and recommend prioritised improvements; track the outcomes of agreed actions.

·       Maintain an accurate inventory of production AI services and assess coverage gaps in telemetry, alerts, runbooks, cost controls, and security guardrails.

Qualifications

Required

·       Bachelor's degree in Computer Science, Engineering, Data Analytics, Artificial Intelligence, or a related discipline, or equivalent relevant experience.

·       3–5 years of relevant experience in IT operations, application support, operational analytics, DevOps, or a related technical area, including hands-on responsibility for monitoring, incident investigation, or service reporting.

·       Practical understanding of AI or machine learning applications, with experience supporting AI-enabled services or analysing their operational data.

·       Working proficiency with scripting language, APIs, and structured data analysis, including automation of routine operational tasks.

·       Hands-on familiarity with cloud services, version control, and software delivery processes; ability to work within established testing and change-management practices.

·       Experience using dashboards, logs, metrics, and traces to investigate incidents, explain service trends, and improve operational reporting.

·       Working understanding of generative AI application patterns, including prompts, model responses, RAG, and tool calling, and their implications for monitoring and troubleshooting.

·       Working knowledge of secure data handling, access controls, privacy-aware logging, and operational audit evidence.

·       Ability to independently manage assigned operational activities, investigate ambiguous issues, recommend improvements, and recognise when specialist escalation is required.

·       Clear technical communication, effective cross-team coordination, and strong documentation and follow-through skills.

Preferred

·       Experience in regulated environments with security, privacy, quality, risk, or audit requirements.

·       Experience supporting an AI application, RAG workflow, chatbot, or agent framework in a live service environment.

·       Experience with consumption reporting, cloud cost analysis, dashboards, and tracking the impact of optimisation actions.

·       Experience with incident and problem management, service-level objectives, deployment-readiness checks, CI/CD, or infrastructure-as-code workflows.

Required Skills:

Preferred Skills:

AI Judgement Integration, Analytical Reasoning, Coaching, Communication, Cross-Functional Collaboration, Demand Forecasting, Human-Computer Interaction (HCI), Persistence and Tenacity, Product Costing, Product Development, Product Strategies, Quality Assurance (QA), Research and Development, Researching, Software Development Management, Technical Credibility, Technologically Savvy

The anticipated base pay range for this position is:

€40.600,00 - €65.320,00

Benefits:

In addition to base pay, we offer the following benefits*: an annual bonus with set target (% of pay) depending on pay grade / location, where the actual amount is based on the employees’ and companies’ performance of the previous calendar year, or sales commissions. Moreover, we offer vacation days, parental leave for a minimum of 12 weeks, bereavement leave, caregiver leave, volunteer leave, well-being reimbursement, programs for financial, physical and mental health. We also offer service anniversary and recognition awards, and subject to the terms of their respective plans, employees - and in some location’s eligible dependents - can participate in several insurance plans. For more information, visit Employee benefits | Supporting well-being & career growth | Johnson & Johnson Careers.

*This is for informative purposes only. Amounts and actual benefits may vary by location and are subject to change.

Additional information:

The anticipated base pay range for this position in SPAIN is EUR 43.600 to EUR 70.150
The anticipated base pay range for this position in PORTUGAL is EUR 33.100 to EUR 52.670