Senior Manager, AI Reliability Engineering

PLEASE NOTE:  This role is on the Kroger Technology & Digital (KTD) team.  KTD is the technology division of The Kroger Co. responsible for building, operating, and innovating the digital infrastructure and software that powers Kroger’s retail stores, e-commerce platforms, supply chain, and corporate operations.  There is a strong collaboration between KTD and 84.51°.

Role Summary 
The Senior Manager, AI Reliability Engineering at KTD leads the engineering discipline that makes enterprise AI operationally trustworthy at scale. As the enterprise moves from AI pilots to production systems that associates and customers depend on every day, this leader ensures that models, agents, copilots, AI gateways and shared runtime services are dependable, high-quality, responsive and cost-efficient by design.  This leader will stand up a new discipline from the ground up: defining what production-grade AI means, engineering the standards and automation that bake reliability and quality into every system, and shaping how build teams design for resilience from day one. The role blends engineering leadership, technical ownership and cross-functional influence. Reliability is a strategic enabler of adoption and velocity - the difference between experimenting with AI and confidently running it at scale. 

Key Responsibilities 
*Build the Discipline (0-to-1) 

*Define what production-grade, operationally trustworthy AI means for the enterprise, including standards and quality bars for availability, behavior, latency, cost, control and recovery. 

*Stand up the AI Reliability Engineering function, its charter, operating model, roadmap, talent model and engineering culture. 

*Position reliability as an enabler of AI adoption and velocity, creating the confidence that allows the business to scale AI responsibly and aggressively. 

*Engineer Reliability and Quality Into Systems 

*Partner with Platform, Model and Applied AI teams to embed resilience, testability, observability and safe failure modes into AI systems from architecture forward. 

*Build reliability tooling and automation, including self-healing, automated evaluations, quality-regression detection, guardrail instrumentation and safe deployment controls. 

*Establish service-level indicators, service-level objectives, error budgets and reliability scorecards that shape architecture, delivery and roadmap decisions. 

*Engineer for high availability, graceful degradation, capacity, disaster recovery and rapid restoration while reducing manual toil and systemic failure patterns. 

*Provide Production Readiness and Agent Onboarding

*Create production-readiness standards covering named business and engineering owners, support models, runbooks, telemetry, quality evaluations, security and Responsible AI controls, escalation paths, service objectives and lifecycle controls. 

*Lead launch-readiness reviews for new platforms, models and agents, and make evidence-based readiness decisions with clear exception and risk-acceptance paths. 

*Build a scalable onboarding model for both centrally developed agents and domain-owned agents operating on shared enterprise platforms. 

*Own Production Quality, Observability and AgentOps 

*Own live observability, production-quality signals and leadership visibility across model and agent behavior, drift, hallucination and quality rates, latency, tool failures and evaluations in production. 

*Partner with Responsible AI to translate offline evaluation, risk and safety standards into continuous, automated production signals and operational controls. 

*Own the operational capabilities of the Agent Control Center, including estate health, pause, isolation, rollback, shutdown and lifecycle controls for unsupported or persistently unreliable agents. 

*Drive Efficiency and Performance 

*Engineer for cost and performance at scale, optimizing inference cost, token efficiency, model and routing economics, tool usage and infrastructure consumption as first-class objectives. 

*Partner with value tracking, metering, product and finance teams to translate consumption and capacity signals into investment decisions and surface the reliability, quality, latency and cost tradeoffs that shape AI strategy. 

*Provide resilience and incident excellence 

*Establish agentic incident prevention and response practices, including severity frameworks, on-call, escalation, incident command, runbooks, communications and blameless learning. 

*Own detection, initial diagnosis, containment, recovery and coordinated L1/L2 engineering response; route L3 defects to the accountable platform, model or agent engineering team. 

Continuously improve detection, recovery, recurrence and deployment safety through automation, permanent engineered fixes, release gates, controlled rollout, automated rollback and recovery testing. 

*Lead and Influence 

*Hire, coach and grow a high-caliber multidisciplinary team spanning site reliability, observability, AgentOps, release engineering and production readiness, with a culture grounded in ownership, automation and blameless learning. 

*Operate horizontally across Platform Engineering, Applied AI, Models, Gateway, Infrastructure, Security and Responsible AI, elevating production engineering across all of them. 

*Act as a technical thought leader for production-grade AI and represent production reality in enterprise architecture, strategy and roadmap decisions. 

Required Qualifications: 
*Minimum 10+ years of experience in software, platform, cloud, SRE or production engineering, including 3 or more years leading engineering teams. 

*Track record of building engineering capabilities, standards, automation or platforms that measurably improved system quality and reliability at scale. 

*Strong technical depth in distributed systems, Kubernetes, cloud infrastructure on GCP and/or Azure, networking, identity, automation, CI/CD and modern observability engineering. 

*Experience establishing SLOs, error budgets, on-call practices, production-readiness reviews, incident management and post-incident improvement programs. 

*Direct experience with ML, AI or LLM systems in production, or demonstrated ability to master failure modes unique to models and agents, including drift, non-determinism, quality regression, prompt and context issues, and hallucination. 

*Ability to influence architecture and roadmaps, lead through ambiguity, partner across a matrixed organization and drive outcomes without direct authority. 

*Excellent technical and executive communication, with the ability to shape strategy with leaders and go deep with senior engineers. 

Preferred Qualifications: 
*Hands-on experience with LLMOps, model observability and evaluation tooling such as LangSmith, MLflow, Arize, Fiddler or custom evaluation pipelines. 

*Familiarity with agentic systems, including orchestration, tool use, MCP-based integrations, registries and their unique reliability and quality challenges. 

*Background bridging software or SRE practices with ML, data science and Responsible AI engineering cultures, including experience establishing a new engineering discipline. 

*Experience with multi-region, high-availability services, hybrid operating models and regulated or high-risk enterprise workloads. 

Measures of Success: 
*SLO attainment, service availability and production quality of AI platforms, models and agents. 

*Mean time to detect, contain and recover from incidents. 

*Production-onboarding time, production-readiness compliance and coverage of ownership, telemetry, runbooks, evaluations and shutdown controls. 

*Quality-regression detection and containment, change-failure, rollback and repeat-incident rates. 

*Reduction in operational toil, noisy alerts and manual interventions. 

*Cost, capacity, latency and token efficiency of production AI workloads. 

*Strength, engagement and growth of the AI Reliability Engineering team. 

#LI-SSS