Observability Engineer - Observability Engineering

Function: Observability Engineering / SRE / Cloud Engineering
 

About the Role

We are looking for an Observability Engineer to join our Observability Engineering team and help build, standardize, and operate a modern enterprise observability platform across cloud and application environments.

The ideal candidate will have strong hands-on engineering experience with New Relic, Grafana, and modern observability technologies, combined with solid foundations in cloud engineering, automation, DevOps, and SRE practices.

This role goes beyond dashboard creation and monitoring operations. You will be responsible for engineering scalable observability solutions, defining telemetry standards, building reusable monitoring capabilities, improving application and infrastructure visibility, and enabling engineering teams to adopt observability as part of their software delivery lifecycle.

You will work closely with SRE, Cloud Engineering, Platform Engineering, Application Engineering, and Operations teams to build a consistent and scalable observability experience across the organization.

Key Responsibilities

1. Observability Engineering

• Design, implement, and maintain enterprise-grade observability solutions across applications, infrastructure, cloud platforms, and services.
• Build and maintain monitoring, alerting, dashboards, service health views, and operational telemetry.
• Develop standardized observability patterns for metrics, logs, traces, events, and application performance monitoring.
• Implement observability solutions using New Relic, Grafana, and other industry-standard tools.
• Develop reusable dashboards, alerts, instrumentation patterns, and observability components.
• Establish observability standards and best practices across engineering teams.
• Continuously improve signal quality by reducing alert noise, false positives, and non-actionable alerts.

2. New Relic Engineering

• Hands-on engineering experience with New Relic APM, Infrastructure Monitoring, Browser Monitoring, Synthetic Monitoring, Logs, Distributed Tracing, NRQL, Alerts, Workloads and Dashboards.
• Design and implement New Relic monitoring and alerting strategies for enterprise applications.
• Develop complex NRQL queries, alert conditions, dashboards, and operational views.
• Configure and optimize New Relic agents and integrations.
• Implement application and infrastructure instrumentation.
• Develop reusable New Relic configurations and automation using APIs/IaC where appropriate.
• Participate in New Relic platform governance, licensing optimization, and standardization.
• Evaluate and implement emerging New Relic capabilities to improve engineering productivity and reliability.

3. Grafana & Visualization

• Build and maintain operational dashboards using Grafana.
• Integrate Grafana with multiple telemetry and data sources.
• Design effective dashboards for application health, infrastructure, SRE, NOC, and executive operational visibility.
• Develop visualization standards and reusable dashboard templates.
• Understand the difference between visualization, monitoring, alerting, and observability, and apply each appropriately.

4. OpenTelemetry & Modern Observability

• Experience with OpenTelemetry and modern telemetry architectures.
• Implement and manage telemetry collection for metrics, logs, and traces.
• Understand distributed tracing and service dependency mapping.
• Work with telemetry pipelines, collectors, agents, exporters, and integrations.
• Experience with technologies such as Prometheus, Loki, Elastic, Splunk, Datadog, Dynatrace, AppDynamics, or similar observability platforms is desirable.
• Evaluate new observability technologies and recommend solutions based on scalability, cost, reliability, and engineering value.

5. Cloud Engineering

Strong cloud engineering fundamentals are expected, including experience with one or more major cloud platforms:

• AWS
• Microsoft Azure
• Google Cloud Platform

Experience should include:

• Compute, networking, storage, databases, containers, and cloud-native services.
• Cloud monitoring and logging.
• IAM and security fundamentals.
• Infrastructure automation.
• Cloud-native architecture and operational best practices.
• Troubleshooting distributed cloud environments.

6. Automation & Infrastructure as Code

• Automate repetitive observability and operational activities.
• Develop scripts and tools using Python, Bash, Go, or similar languages.
• Use Terraform / OpenTofu or equivalent Infrastructure as Code technologies.
• Build reusable automation for dashboards, alerts, instrumentation, integrations, and configuration management.
• Integrate observability capabilities into CI/CD pipelines.

7. DevOps & CI/CD

• Experience with modern CI/CD practices and tools.
• Hands-on experience with GitHub Actions, Jenkins, GitLab CI, Azure DevOps, or similar platforms.
• Integrate observability and quality gates into deployment pipelines.
• Implement deployment markers and release health monitoring.
• Enable automated validation of application and infrastructure health following deployments.

8. SRE & Reliability Engineering

• Apply SRE principles to improve system reliability and operational maturity.
• Define and monitor SLIs, SLOs, and error budgets.
• Participate in incident investigation and root-cause analysis.
• Develop proactive monitoring and reliability solutions.
• Identify reliability gaps and engineer solutions to eliminate recurring incidents.
• Support capacity, performance, availability, and resilience engineering.

Required Skills & Experience

Must Have

• 5+ years of experience in Observability, SRE, DevOps, Cloud Engineering, Platform Engineering, or a related engineering discipline.
• Strong hands-on experience with New Relic.
• Strong hands-on experience with Grafana.
• Experience building production-grade dashboards, monitoring and alerting solutions.
• Strong understanding of APM, infrastructure monitoring, logging, metrics, tracing, and distributed systems.
• Experience with NRQL and New Relic alerting.
• Experience with OpenTelemetry is highly desirable.
• Strong cloud engineering experience in AWS, Azure, or GCP.
• Strong scripting/programming experience in Python, Bash, Go, or similar.
• Experience with Terraform/OpenTofu or another Infrastructure-as-Code technology.
• Understanding of CI/CD and DevOps practices.
• Understanding of SRE principles, SLIs, SLOs, incident management, and reliability engineering.
• Strong troubleshooting and analytical skills.

Good to Have

• New Relic certifications or equivalent hands-on expertise.
• Grafana/Prometheus experience.
• OpenTelemetry implementation experience.
• Kubernetes and container observability.
• Experience with Prometheus, Loki, Elastic, Splunk, Datadog, Dynatrace, AppDynamics or similar platforms.
• Experience designing enterprise observability architectures.
• Experience with observability platform migrations or consolidation.
• Experience with observability cost optimization and licensing governance.
• Experience developing observability-as-code.
• Experience integrating observability with ServiceNow, Jira, PagerDuty, Opsgenie, or similar ITSM/incident platforms.
• Experience with AI-assisted observability, AIOps, anomaly detection, or automated incident investigation.

What Success Looks Like

In this role, you will be successful when you can:

• Engineer rather than simply operate monitoring.
• Build scalable observability solutions that can be reused across hundreds of applications.
• Turn raw telemetry into meaningful engineering and operational insights.
• Reduce alert noise and improve signal quality.
• Standardize New Relic and Grafana adoption across engineering teams.
• Automate observability configuration and reduce manual operational work.
• Improve application reliability through better telemetry and proactive detection.
• Enable engineering teams to own more of their operational health.
• Help establish Observability as a Platform/Capability, rather than simply another monitoring tool.

Behavioral & Leadership Skills

• Strong ownership and accountability.
• Ability to work across engineering, application, infrastructure, and operations teams.
• Strong communication and stakeholder management skills.
• Ability to explain complex technical concepts to both engineers and non-technical stakeholders.
• Strong problem-solving and analytical mindset.
• Comfortable working in a fast-paced, enterprise engineering environment.
• Passion for automation, engineering excellence, reliability, and continuous improvement.
• Ability to challenge existing approaches and introduce better engineering practices.