Principal Core Infrastructure Engineer

Oracle Cloud Infrastructure (OCI) is seeking a Site Reliability Engineer to operate and improve the services that manage SmartNICs across our cloud infrastructure. You will help ensure the reliability of card provisioning, configuration, deployment, health monitoring, and recovery throughout the hardware lifecycle.

This role combines software engineering, systems troubleshooting, and production operations. You will work closely with software, networking, firmware, hardware, and data center teams to keep a large distributed fleet healthy and available.

• Operate and improve services responsible for SmartNIC discovery, provisioning, configuration, upgrades, and decommissioning.
• Define and track service-level objectives for management services and fleet operations.
• Build observability for card health, lifecycle transitions, firmware rollout progress, configuration consistency, and failed operations.
• Troubleshoot issues across management services, Linux systems, networking, firmware, and hardware interfaces.
• Develop automation to detect unhealthy cards, collect diagnostics, and perform safe remediation.
• Improve firmware and software deployments through staged rollouts, health validation, rollback mechanisms, and controlled concurrency.
• Design reliable lifecycle workflows that handle retries, partial failures, timeouts, and interrupted operations.
• Participate in on-call rotations, respond to production incidents, and drive corrective actions through blameless incident reviews.
• Partner with engineering teams to address recurring failures and improve serviceability, resilience, and operational readiness.
• Maintain runbooks and recovery procedures, and validate them through failure testing and operational exercises.
• Plan for fleet growth, management-service capacity, and operational scalability.
  
  Required Qualifications
• Experience in site reliability engineering, systems engineering, production engineering, or distributed infrastructure operations.
• Strong Linux troubleshooting skills, including processes, memory, storage, networking, and system logs.
• Programming experience in Python, shell scripts or another language used to build production services and automation.
• Understanding of distributed systems, APIs, asynchronous workflows, and failure recovery.
• Knowledge of networking fundamentals, including TCP/IP, routing, switching, and packet-level troubleshooting.
• Experience with monitoring, alerting, incident response, and root-cause analysis.
• Ability to investigate problems that span software and hardware boundaries.
• Clear communication and effective coordination during production incidents.

Preferred Qualifications

• Experience with SmartNICs, DPUs, network interface controllers, or hardware management platforms.
• Familiarity with firmware lifecycle management, device boot sequences, PCIe, drivers, and out-of-band management.
• Experience operating large hardware fleets in cloud or data center environments.
• Understanding of network virtualization, bare-metal infrastructure, and control-plane services.
• Experience designing safe fleet-wide changes with rate limits, canary deployments, and failure containment.
• Familiarity with hardware diagnostics, telemetry, and coordination with data center operations.