Senior Network Operations Engineer
Cotidal
Senior Network Operations Engineer
Cotidal
AI-focused supercomputing infrastructure provider
Supercomputing built for AI.
Cotidal is building the computing infrastructure that AI companies and research labs need to train and run their models. Starting with TPUs, we design the hardware and software together around their workloads, with the goal of making powerful AI chips easier to use at scale.
Our mission is endless compute for humanity. Over time, we want customers to choose across chip stacks without rebuilding their infrastructure.
We’re hiring a Senior Network Operations Engineer to keep Cotidal’s data center networks and backbone available, reliable and performing well. You will operate and troubleshoot production networks, lead incident response, and turn what you learn into better monitoring, automation and operating procedures.
You will bring hands-on production experience with Juniper and Arista routing and switching, Palo Alto and Juniper firewalls, and high-performance Ethernet fabrics supporting AI/HPC and GPU workloads. You will work across network, compute, storage and platform engineering, with responsibility for diagnosing complex failures and carrying fixes through to production.
Operate data center fabrics and backbone networks. Manage Juniper MX, QFX and EX platforms and Arista EOS switches. Troubleshoot BGP, OSPF, IS-IS, MPLS, EVPN/VXLAN and ECMP across AI/HPC network environments.
Keep high-performance fabrics healthy. Support RDMA/RoCEv2 operations, including PFC, ECN and QoS. Investigate congestion, packet loss, latency and performance degradation in AI/HPC and GPU cluster networks.
Operate network security infrastructure. Maintain Palo Alto next-generation firewalls and Juniper SRX environments, including security policies, NAT, IPsec VPNs and high availability.
Lead production incident response. Diagnose difficult failures, manage operational escalations, lead root-cause analysis and follow through on corrective actions.
Make safe changes and watch their impact. Monitor telemetry, interface errors, optics, utilization and network health across 100G/400G/800G infrastructure. Carry out software upgrades, maintenance, capacity expansions and deployments through change control.
Build repeatable operations. Develop automation, monitoring dashboards, runbooks and operational documentation, and participate in on-call support.
10–15 years of professional network engineering and operations experience.
Strong production experience with Junos and Juniper MX/QFX/EX routing and switching, plus hands-on Arista EOS operations.
Advanced troubleshooting of BGP, OSPF, IS-IS, MPLS, EVPN/VXLAN, ECMP, TCP/IP, IPv4/IPv6, LAG/LACP and spine-leaf fabrics.
Working knowledge of GPU cluster networking, AI/HPC networking, RDMA/RoCEv2, PFC, ECN, QoS and congestion management. You can isolate packet drops, microbursts, high latency and performance degradation.
Hands-on Palo Alto and Juniper SRX firewall operations, including policy troubleshooting, NAT, VPNs, high availability and segmentation.
Experience with SNMP, syslog, streaming telemetry, packet captures, Wireshark/tcpdump and Linux networking.
Familiarity with Python, Ansible, Git, incident management, root-cause analysis and change control.
The ability to resolve complex production incidents independently and communicate clearly across teams.
Experience in one or more of the following:
Operating hyperscale, cloud or AI/HPC environments, including NVIDIA Spectrum Ethernet, InfiniBand or AMD GPU infrastructure.
An Optical Circuit Switching (OCS) layer, including optical circuit provisioning, path management, monitoring, fault isolation and integration with data center fabrics.
NVIDIA NCCL, distributed GPU workloads and RDMA performance tuning.
SONiC, Prometheus, Grafana or Splunk.
Juniper JNCIP/JNCIE, Palo Alto PCNSE or equivalent certifications.
Help establish how we run the networks that connect our AI infrastructure. Your work will link production reliability, performance and safe expansion, with the opportunity to build operating practices that future deployments can use.
Help shape the foundations of an early company through direct discussion and work across network, systems, storage and platform engineering.
Location: Palo Alto, in person.
On-call: Participate in production network on-call support.
Compensation: Competitive salary and equity.
Visa sponsorship: We sponsor work visas and help you navigate the process.
Benefits: Medical, dental, and vision coverage, unlimited PTO, and relocation support as needed.
Cotidal builds specialized AI compute infrastructure, creating supercomputers for training and running large AI models for leading AI companies and research labs. Its approach pairs hardware and software in a single design, engineering silicon, code, and data-center deployment together to maximize performance for AI workloads; initially it centers on TPUs with a goal of supporting multiple chip stacks later. This holistic co-design differentiates Cotidal from vendors that sell hardware or software separately, aiming for hardware-agnostic efficiency as chip tech evolves. Its goal is to speed breakthroughs in AI by providing scalable, integrated compute for fields like medicine, physics, and robotics.
Explore more opportunities at Cotidal