Principal Site Reliability Engineer

Position Overview:
In this role, you will focus on ensuring the availability, reliability, and performance of our multi-tenant, microservices application suite. You will collaborate closely with cross-functional teams to troubleshoot issues, automate processes, and build scalable, resilient systems. You will learn the nuances of the entire KRADOS suite of applications and its infrastructure, which will facilitate your missions of 24/7/365 tier 2/3 outage response, and improving the efficiencies of KRADOS.

Key Responsibilities:

• Monitor system health, define Service Level Indicators (SLIs), and ensure adherence to Service Level Objectives (SLOs).
• Respond promptly to outages, conduct root cause analyses, and implement durable solutions to prevent recurrence.
• Collaborate with development and DevOps teams to optimize and maintain Kubernetes environments and CI/CD pipelines.
• Develop and refine automation scripts to enhance system reliability, including automated recovery and self-healing capabilities.
• Build and maintain observability frameworks, integrating metrics, logging, and tracing tools for proactive issue identification.
• Contribute to performance tuning and scalability improvements across the application stack.
• Document incident responses and contribute to a knowledge base to foster a culture of continuous improvement.
• Participate in an on-call rotation to provide 24/7/365 support for mission-critical systems.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.
• 6 years of experience in site reliability, systems engineering, or DevOps roles.
• Proficiency in one or more programming/scripting languages (e.g., Python, Go, Java, Bash).
• Strong understanding of distributed systems, microservices architecture, and RESTful API design.
• Hands-on experience with Kubernetes and container orchestration.
• Familiarity with monitoring, alerting, and logging tools (e.g., Prometheus, Grafana, ELK stack, or Datadog). Experience with Elastic will be highly helpful with this position.
• Hands-on experience with incident response, including designing and improving incident management processes.
• Expertise in Observability practices, including metrics, logs, traces, and understanding of distributed tracing tools (e.g., OpenTelemetry).
• Strong problem-solving skills with a focus on building resilient, fault-tolerant systems.
• Excellent communication skills and a collaborative mindset.
• Have to have SEC+ or higher certification or ability to obtain it within six months from hire.
• Must be willing to do shift work to provide 24/7/365 coverage.
• Must be within 45 minutes drive of an IL6 workstation location (e.g., SIPR café, SCIF)

Preferred Qualifications:

• Experience with cloud platforms (e.g., AWS) and their associated managed services.
• Knowledge of database management and optimization for systems (e.g., PostgreSQL)
• Familiarity with Infrastructure as Code (IaC) tools (e.g., Terraform, CloudFormation).
• Master’s degree in Computer Science, Engineering, or a related field, or equivalent practical experience.