BMC Software

IT management software for mainframes

Lead Site Reliability Engineer

Site Reliability EngineeringConfirmed live in the last 24 hours

Full-time

About the Job

Requirements

Foundational site reliability engineering knowledge, including service-level objectives and indicators, monitoring, alerting, and incident management.
Knowledge of Linux, networking, distributed systems, troubleshooting, and performance analysis.
Knowledge of continuous integration and continuous delivery, DevSecOps, Docker, Kubernetes, OpenShift, and cloud platforms.
Generative AI and large language model literacy, with knowledge of retrieval-augmented generation, agents, and tooling as appropriate to level.
MLOps/LLMOps tooling and AI observability knowledge as appropriate to level.
Python and Bash scripting and automation skills.
Linux and scripting fundamentals, with exposure to monitoring or cloud; project or internship experience is acceptable.
Curiosity about how large language model and agent systems fail in production.
Ability to work calmly and carefully under guidance and readiness for eventual on-call work with support.
Experience contributing to monitoring, runbooks, or operational fixes within an existing workflow, with basic exposure to cloud or containers.
Evidence of reliable execution of defined operations tasks, clear notes, and careful changes under guidance.
Ability to complete defined tasks with guidance and check work carefully.

Responsibilities

Build and operate production-grade infrastructure and operational frameworks for large language model, generative AI, retrieval-augmented generation, and agentic AI applications.
Define and defend service-level objectives and indicators for agent and model behavior, including latency, availability, quality, and cost.
Monitor production AI systems for performance, drift, hallucination rates, and quality regressions, and act before customers are affected.
Lead incident response for AI systems, including detection, triage, mitigation, and blameless post-mortems.
Build runbooks, automation, and self-healing for common AI operational issues, and participate in on-call as required.
Operate model and agent lifecycles in production, including versioning, rollout, rollback, and safe promotion with evaluation gates.
Manage large language model cost, capacity, scaling, and model-serving/inference infrastructure.
Maintain observability and tracing for agents, tools, and prompts using Langfuse, OpenTelemetry, and OpenSearch.
Operate AI/ML workloads on IBM Z where required and harden the AMI Platform for reliability, scalability, and security.
Partner with AI engineers, data scientists, DevOps, and security teams to embed reliability from design onward.
Complete defined operations and observability tasks with guidance, learn AI failure modes and on-call basics, and contribute reliable work to a project.

Desired Qualifications

Experience with agent frameworks and AgentOps for multi-agent systems.
Experience with large language model cost optimization and inference-serving stacks such as vLLM and Ollama.
Experience with enterprise on-premises or hybrid deployments and Agile/Atlassian.

CareerArc Code

CA-SW

Hybrid: #LI-Hybrid

BMC empowers nearly 80% of the Forbes Global 100 to accelerate business value, faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage.

About the Role

You build, operate, and continuously improve reliable, scalable, secure, and observable AI/ML and Generative AI platforms in production — supporting the full lifecycle from deployment and monitoring through incident management, optimization, and continuous improvement.

At this level you execute defined work with guidance and grow through review and mentorship.

Scope at this level: Completes defined operations/observability tasks with guidance; learns AI failure modes and on-call basics.

Organizational impact expected: Contributes reliable work to a project.

Key Responsibilities

Core responsibilities for this role:

• Build and operate production-grade infrastructure and operational frameworks for LLM, GenAI, RAG, and Agentic AI applications.

• Define and defend SLOs/SLIs for agent and model behaviour — latency, availability, quality, and cost.

• Monitor production AI systems for performance, drift, hallucination rates, and quality regressions; act before customers are affected.

• Lead incident response for AI systems: detection, triage, mitigation, and blameless post-mortems.

• Build runbooks, automation, and self-healing for common AI operational issues; participate in on-call as required.

• Operate model/agent lifecycle in production: versioning, rollout, rollback, and safe promotion with evaluation gates.

• Manage LLM cost, capacity, and scaling; manage model serving/inference infrastructure.

• Maintain observability/tracing for agents, tools, and prompts (Langfuse, OpenTelemetry, OpenSearch).

• Operate AI/ML workloads on IBM Z where required; harden the AMI Platform for reliability, scalability, and security.

• Partner with AI Engineers, Data Scientists, DevOps, and Security to embed reliability from design onward.

Must-Have Skills & Experience

· SRE fundamentals — SLOs/SLIs, monitoring, alerting, incident management (depth scales with level).

· Linux, networking, distributed systems, troubleshooting, performance analysis.

· CI/CD and DevSecOps practices; containers/orchestration (Docker, Kubernetes, OpenShift); cloud platforms.

· Generative AI/LLM literacy; RAG/agents/tooling concepts as level requires.

· MLOps/LLMOps tooling and AI observability as level requires.

· Scripting/automation (Python, Bash); security-first, calm incident handling.

· Linux/scripting fundamentals; exposure to monitoring or cloud (project/internship OK).

· Curiosity about how LLM/agent systems fail in production.

· Calm, careful work under guidance; readiness for eventual on-call with support.

· Past experience: Contributed to monitoring, runbooks, or operational fixes within an existing workflow; basic exposure to cloud/containers.

· Delivery evidence: Reliable execution of defined ops tasks; clear notes; careful changes under guidance.

· Shared expectation: Completes defined tasks with guidance and checks work carefully.

Nice-to-Have Skills

· Agent frameworks and AgentOps for multi-agent systems.

· LLM cost-optimization and inference-serving stacks (vLLM, Ollama).

· Enterprise on-prem or hybrid deployments; Agile/Atlassian.

#LI-Remote

Our commitment to you!

BMC’s culture is built around its people. We have 6000+ brilliant minds working together across the globe. You won’t be known just by your employee number, but for your true authentic self. BMC lets you be YOU!

If after reading the above, You’re unsure if you meet the qualifications of this role but are deeply excited about BMC and this team, we still encourage you to apply! We want to attract talents from diverse backgrounds and experience to ensure we face the world together with the best ideas!

BMC is committed to equal opportunity employment regardless of race, age, sex, creed, color, religion, citizenship status, sexual orientation, gender, gender expression, gender identity, national origin, disability, marital status, pregnancy, disabled veteran or status as a protected veteran. If you need a reasonable accommodation for any part of the application and hiring process, visit the accommodation request page.

BMC Software maintains a strict policy of not requesting any form of payment in exchange for employment opportunities, upholding a fair and ethical hiring process.

The annual base salary range represents the low and high end of the BMC salary range for this position. Actual salaries depend on a wide range of factors that are considered in making compensation decisions, including but not limited to skill sets; experience and training, licensure, and certifications; and other business and organizational needs.

The range listed is just one component of BMC's employee compensation package. Other rewards may include a variable plan and country specific benefits.

At BMC, it is not typical for an individual to be hired at /near the top of the range. A reasonable estimate of the current range is $152,925 - $254,875

Min salary

152,925

Mid point salary

203,900

Max salary

254,875

Min Salary - NEW

152,925

Max Salary - NEW

254,875

About the Company

BMC Software

IT management software for mainframes

What BMC Software does: It provides software to manage and optimize large enterprise IT environments, from IBM mainframes to modern cloud systems. How its product works: Its tools monitor, manage, and automate IT infrastructure across mainframes, distributed servers, and cloud platforms, helping teams keep systems running, secure, and efficient. How it stands out from competitors: It has a long track record starting with mainframe software and has grown through targeted acquisitions to offer a broad suite for end-to-end IT operations, spanning on‑premises and cloud, with strategic ownership changes that gave it room to adapt. What its goal is: to help enterprises operate complex IT environments more reliably and efficiently by providing integrated software for mainframes, distributed systems, and cloud management.

Company size: 5,001-10,000
Company stage: IPO
Headquarters: Houston, TX
Founded: 1980

Recently Posted Jobs

Explore more opportunities at BMC Software

See more jobs