Lead Principal Site Reliability Engineer

Key Responsibilities

Reliability Strategy and Technical Leadership

• Define and drive the site reliability engineering strategy for large-scale, distributed, and business-critical platforms. 
• Establish reliability standards, engineering practices, and operational readiness requirements across multiple teams. 
• Serve as a senior technical authority for system reliability, scalability, resilience, performance, and production operations. 
• Influence architecture and design decisions to ensure systems are supportable, observable, fault tolerant, and capable of meeting availability objectives. 
• Identify systemic reliability risks and lead cross-functional initiatives to address them. 
• Provide technical direction and mentorship to site reliability engineers, software engineers, platform engineers, and operations teams. 
• Lead technical reviews and promote consistent engineering practices across the organization. 

Service Reliability and Observability

• Define and implement service-level indicators, service-level objectives, error budgets, and operational health metrics. 
• Develop comprehensive monitoring, logging, tracing, alerting, and observability strategies. 
• Improve the quality and actionability of alerts while reducing unnecessary operational noise. 
• Establish dashboards and reporting mechanisms that clearly communicate service health, performance, capacity, and risk. 
• Use production data and reliability trends to prioritize engineering investments and continuous-improvement initiatives.  

Automation and Platform Engineering

• Design and implement automation that reduces manual intervention, operational toil, and human error. 
• Build or enhance tools for deployment, configuration management, infrastructure provisioning, incident response, and service recovery. 
• Promote infrastructure-as-code, policy-as-code, automated testing, and repeatable deployment practices. 
• Partner with development teams to improve continuous integration and continuous delivery pipelines. 
• Develop self-healing and automated remediation capabilities where appropriate. 
• Contribute production-quality software and reusable platform components using modern programming and scripting languages. 

Incident Management and Problem Resolution

• Provide technical leadership during complex, high-severity production incidents. 
• Coordinate diagnosis, containment, recovery, and stakeholder communication during service disruptions. 
• Lead blameless post-incident reviews and ensure that corrective actions address root causes rather than symptoms. 
• Identify recurring failure patterns and develop long-term engineering solutions. 
• Improve incident-management processes, escalation procedures, runbooks, and recovery playbooks. 
• Participate in an on-call rotation or provide senior escalation support for critical services, as required. 

Capacity, Performance, and Resilience

• Lead capacity planning, performance analysis, load testing, and demand forecasting for critical platforms. 
• Identify performance bottlenecks and recommend architectural or operational improvements. 
• Design and validate high-availability, disaster-recovery, backup, and business-continuity capabilities. 
• Lead resilience testing, failure-mode analysis, game days, and controlled fault-injection exercises. 
• Ensure recovery-time and recovery-point objectives are defined, tested, and achievable. 

Security and Operational Governance

• Partner with security and compliance teams to embed security into infrastructure, automation, and operational practices. 
• Support vulnerability remediation, access-control improvements, audit readiness, and secure configuration management. 
• Ensure production environments meet organizational standards for change management, data protection, and operational governance. 
• Balance reliability, security, delivery speed, cost, and business priorities when recommending technical solutions. 

Required Qualifications

• Extensive professional experience in site reliability engineering, software engineering, cloud infrastructure, platform engineering, systems engineering, or a related technical discipline. 
• Demonstrated experience designing, operating, and improving highly available production systems at significant scale. 
• Deep knowledge of distributed systems, cloud architecture, networking, operating systems, storage, databases, and service dependencies. 
• Advanced experience with at least one major cloud platform, such as Oracle Cloud Infrastructure, Amazon Web Services, Microsoft Azure, or Google Cloud Platform. 
• Strong experience with containerization and orchestration technologies, including Docker and Kubernetes. 
• Proven expertise with infrastructure-as-code and configuration-management technologies such as Terraform, Ansible, Chef, Puppet, or equivalent tools. 
• Experience implementing observability solutions using metrics, logs, traces, dashboards, and automated alerting. 
• Strong programming or scripting skills in one or more languages such as Python, Go, Java, JavaScript, Bash, or similar. 
• Experience with continuous integration, continuous delivery, automated testing, and modern release-management practices. 
• Demonstrated leadership during critical production incidents and complex technical investigations. 
• Ability to diagnose difficult system issues across applications, infrastructure, networks, databases, and cloud services. 
• Strong written and verbal communication skills, including the ability to explain technical risk and recommendations to engineering leaders and business stakeholders. 
• Proven ability to lead cross-functional technical initiatives without relying solely on formal authority. 

Preferred Qualifications

• Experience supporting enterprise-scale cloud services, software-as-a-service platforms, or other high-availability customer-facing systems. 
• Experience defining and operating service-level objectives, error budgets, and reliability scorecards. 
• Knowledge of chaos engineering, resilience testing, and automated recovery techniques. 
• Experience with multi-region, hybrid-cloud, or multi-cloud architectures. 
• Familiarity with security frameworks, compliance requirements, and regulated operating environments. 
• Experience improving cloud cost efficiency, capacity utilization, or infrastructure performance. 
• Contributions to internal engineering standards, technical communities, open-source projects, or industry publications. 
• Bachelor’s or advanced degree in computer science, engineering, information systems, or a related field, or equivalent practical experience.