Senior Software Engineer - Service Observability & Detection

Summary

Imagine what you could do here. At Apple, new ideas have a way of becoming extraordinary products, services, and customer experiences very quickly. Apple Pay and Wallet are at the heart of how millions of people around the world make payments, store credentials, and interact with commerce every day. Behind every tap, every transaction, and every boarding pass is a set of systems that must be fast, reliable, and secure at massive scale. We are looking for a Senior Software Engineer to join the Service Observability & Detection team within Wallet, Payments, and Commerce (WPC). In this role, you will design and build the telemetry foundations, SLO frameworks, and detection systems that give engineering and operations teams deep, real-time visibility into the health of services that hundreds of millions of people depend on every day. You will define what "good" looks like for telemetry at scale — and build systems that ensure we detect failures before our customers do.

Minimum Qualifications

8+ years of experience in software engineering with deep expertise in observability, reliability engineering, or telemetry platform development. Proven track record of leading cross-functional technical initiatives — driving alignment, adoption, and delivery across multiple engineering teams you don't directly manage. Experience building telemetry and observability systems from the ground up — not just configuring tools, but designing instrumentation strategies, telemetry pipelines, and data models that scale to thousands of services and billions of events. Deep hands-on experience with OpenTelemetry — SDK instrumentation, Collector architecture (agent and gateway tiers), processors, exporters, context propagation, and metadata enrichment at scale. Strong understanding of SLOs, SLIs, and error budgets — with hands-on experience defining, implementing, and operating SLO-based alerting in production. Experience designing alerting architectures for complex distributed systems — including burn-rate alerts, multi-signal correlation, intelligent routing, and alert storm prevention. Strong proficiency in at least two of: Kotlin, Java, Go, or Python. Experience with observability platforms such as Datadog, Grafana, Prometheus, or Splunk — including APM, metrics, traces, monitors, and dashboards. Deep understanding of distributed systems concepts including microservices architectures, API gateways, event-driven systems, and service-to-service dependencies. Experience solving hard telemetry problems at scale — sampling strategies, high-cardinality metric management, tail-based sampling, cross-service trace stitching, and pipeline reliability under load. Strong communication and stakeholder management skills — ability to translate complex technical observability concepts into clear strategies for engineering leadership and partner teams. Demonstrated ability to establish org-wide engineering standards and drive adoption through influence, documentation, and tooling rather than mandate. BS or MS in Computer Science, Software Engineering, or a related field, or equivalent professional experience.

Preferred Qualifications

Experience working on payment systems, financial services platforms, or other high-compliance, high-availability domains. Experience building observability platforms or telemetry infrastructure that serves an entire engineering organization — not just a single team. Experience leading technical initiatives across organizational boundaries — building consensus, defining standards, and driving adoption without direct authority. Experience applying AI/ML to operational problems such as anomaly detection, intelligent alert grouping, automated incident correlation, or predictive alerting. Familiarity with incident management workflows and tools — understanding how detection feeds into triage, escalation, and resolution. Experience with Infrastructure-as-Code tools such as Terraform or Pulumi for managing observability resources (monitors, SLOs, dashboards) as code. Experience with service catalog or service registry systems — ownership models, dependency mapping, and health scoring at organizational scale. Experience building systems that serve both engineering teams and SOC/NOC operators — balancing depth of signal with clarity of action.

Description

Apple Pay processes billions of transactions across millions of merchants worldwide. The systems behind Apple Pay, Apple Wallet, Apple Cash, Tap to Pay, and our commerce platforms must meet the highest standards for availability, latency, security, and correctness — with zero tolerance for downtime or data loss. Our team owns the observability and detection layer across all WPC services. We don't just consume telemetry — we define the telemetry strategy, build the instrumentation standards, and design the SLO and alerting architectures that turn raw signals into actionable detection. We work at the intersection of platform engineering, reliability, and operations — building systems that scale across hundreds of services, thousands of APIs, and billions of transactions. This is a senior engineering role within one of Apple's most critical and high-scale service organizations. You will work cross-functionally with multiple product and engineering teams — partnering directly with offering owners, platform teams, and operations — to build observability and detection systems that the entire organization depends on. You will lead by influence, drive adoption across teams you don't manage, and set the telemetry and reliability standards that shape how WPC builds, monitors, and operates services at scale.

Responsibilities

Design and build scalable telemetry pipelines using OpenTelemetry — including SDK instrumentation libraries, Collector deployments (agent and gateway tiers), processors, filters, exporters, and metadata enrichment — optimized for high-throughput, low-latency environments where every dropped span or miscounted metric has real-world consequences. Define and enforce telemetry standards across a large service organization — metric naming conventions, span attribute schemas, context propagation patterns, and cardinality management strategies that keep telemetry useful and cost-effective as the system scales. Architect SLO frameworks from the ground up — designing multi-tier SLO models (service-level, feature-level, journey-level) with proper SLI definitions, error budget policies, and burn-rate alerting that reflects actual customer experience rather than synthetic health. Build intelligent alerting systems that scale — multi-layered alert architectures that prevent alert storms during cascading failures, route signals to the right responders, and provide clear context on customer impact, blast radius, and root cause attribution. Design and implement end-to-end user journey monitoring across distributed microservice architectures — correlating traces, metrics, and logs across API gateways, orchestration layers, partner integrations, and asynchronous event pipelines to provide flow-level visibility. Build and maintain observability-as-code infrastructure — managing monitors, SLOs, dashboards, and service catalog definitions through version-controlled, templatized, and repeatable deployment pipelines. Develop tooling and automation that accelerates observability adoption — instrumentation gap detection, automated service catalog enrichment, telemetry health checks, and self-service onboarding for engineering teams. Solve hard telemetry engineering problems at scale — sampling strategies that preserve signal fidelity, tail-based sampling for error and latency outliers, high-cardinality metric management, cross-service trace stitching, and telemetry pipeline reliability under load. Drive observability coverage across third-party and partner integrations where you don't own the code — designing proxy metrics, synthetic probes, and inferred service detection to eliminate blind spots at system boundaries.