QA Engineer - O2C & Enterprise Billing Testing

New Relic unveils Infrastructure 360 to provide platform engineers end-to-end infrastructure visibility to power reliable AI workloads. New unified observability umbrella unifies visibility into cloud resources, dependencies and configuration changes for governable, secure and reliable AI at scale. New Relic today announced the launch of Infrastructure 360, a comprehensive solution empowering platform engineering, infrastructure and operations teams to drastically reduce incident response times by giving teams an end-to-end view of their cloud resources, dependencies, and configuration changes. With New Relic's existing infrastructure observability capability as its foundation, Infrastructure 360 is built directly on top to deliver a significantly deeper level of information with a dedicated user interface (UI). This provides engineers with the operational foundation to ensure their teams can deploy AI with confidence. Bringing visibility, guardrails, and context to modern infrastructure management To support modern development, platform engineers must balance four critical pillars: velocity, reliability, cost, and security. However, in the AI era, where infrastructure scaling happens at unprecedented speed and complexity, managing these four pillars has become increasingly untenable using legacy tools. As AI-driven workloads rapidly scale and change underlying resources, the risk of unmonitored edits and silent configuration drift grows exponentially. Half of major production outages stem directly from these untracked infrastructure changes. When an incident occurs, teams are often forced to jump between disconnected tools, struggling to answer basic questions about what is running, monitored, or recently changed. To address these challenges, platform engineering must provide a consistent operational foundation with an end-to-end view of the constantly shifting underlying infrastructure without restricting developer agility. Infrastructure 360 brings unified visibility into all cloud resources, coverage, dependencies, and configuration changes. It helps platform engineers quickly see what changed and correlate configuration updates with performance issues. It also discovers AWS and Azure resources and highlights where metrics coverage is missing, helping teams identify monitoring gaps before they become troubleshooting blind spots. With deeper dependency mapping across applications, Kubernetes, networks, load balancers, and gateways, teams can better understand where an issue originates and what it may impact. The result is faster troubleshooting with a much stronger infrastructure context. "Unlike legacy monitoring platforms that force engineers to piece together host data across fragmented product silos, New Relic consolidates infrastructure, applications, and logs into a single, unified database," said New Relic CTO Michael Frendo. "With Infrastructure 360, we leverage this single-source-of-truth architecture to track exact configuration changes and deep cross-stack dependencies in real time. By eliminating the blind spots and data friction common in multi-tool setups, we deliver the rock-solid, high-fidelity telemetry engine required to keep complex production AI applications running with absolute stability and predictability." Key features and benefits of Infrastructure 360 include: * Automated Resource Discovery & Instrumentation gaps: Infrastructure 360 agentlessly scans connected AWS and Azure accounts to map out all active cloud resources, identifying what exists regardless of its monitoring state. It proactively provides instrumentation blind spots with guided onboarding workflows, transforming passive reporting into active remediation of blind spots. * Configuration Change Tracking & Explorer: The solution automatically captures and stores infrastructure change events in AWS, Azure service configurations and Kubernetes object manifests. Engineers can use the Configuration Explorer to view exact property-level before-and-after diffs, instantly correlating application performance spikes with recent deployments. * Advanced App-Infra Topology Mapping: Moving far beyond APM-to-host connections, Infrastructure 360 maps dependencies connecting APM directly to underlying cloud networks, load balancers, application gateways, databases amongst many others, to eliminate production issues. * Enhanced Data Foundation for New Relic Autopilot: The deep relationship and configuration change data gathered by Infrastructure 360 serves as a critical foundation for AI and automation capabilities. When paired with New Relic Autopilot - a set of automated SRE agents that investigates, explains, and helps fix incidents using real-time data from your environment - this rich operational intelligence enables the AI to make significantly more accurate and context-aware decisions. * Unified Workflow and Intelligence: Unified visibility for platform engineering, infrastructure and operations teams provide reliability, cost and security lenses. All active alerts, observability blind spots, config changes across all of infrastructure are accessible through Infrastructure 360.

New Relic introduces AI Evaluation to close the loop across the entire developer-to-production lifecycle with transaction-level business impact. New Relic, the intelligent observability platform, today announced a framework for advanced AI evaluation. New Relic AI Evaluation is a new capability within New Relic AI Observability. Unlike point solutions that evaluate isolated single-LLM calls, it delivers transaction-level business impact insights across the full developer-to-production lifecycle, tracking response quality and behaviour while providing automated, real-time insights into AI guardrail performance. Delivering end-to-end trust: Full-stack visibility built for the AI era As generative AI becomes deeply embedded across enterprise workflows, software is shifting from deterministic code to probabilistic systems that fail in various ways, from subtle hallucinations to unexpected performance shifts when frontier vendors update LLMs. This creates a new observability requirement as traditional application signals no longer tell the whole story. Teams now need to see which model responded, which prompt was used, what tools an agent called, how much the interaction cost, and whether the result was useful. Observing AI in a silo or through best-of-breed point solutions creates dangerous blind spots around upstream and downstream operational impacts while widening the visibility gap between AI developers and production engineering. A single, unified intelligent observability platform built for the AI era solves this by uniting full-stack operational telemetry with real-time AI evaluations - delivering the essential trust layer and end-to-end visibility leaders need to control costs, mitigate security risks, and safely scale enterprise AI. Built natively into the unified New Relic platform, AI Evaluation goes beyond isolated prompt checks to analyse AI performance across the entire application lifecycle - down to the underlying transaction. By uniting full context rather than focusing on a single metric, it embeds quality and security guardrails directly into existing application performance monitoring workflows. Replacing unscalable manual reviews with an asynchronous "LLM-as-a-judge" service, AI Evaluation automatically scans live telemetry to score vulnerabilities like hallucinations, prompt injections, and data leaks. It attaches these probabilistic quality scores as attributes directly to deterministic distributed traces, allowing teams to isolate the exact root cause of a failure, whether in the prompt, vector database, or backend infrastructure, within a single view. By linking qualitative response scores directly to underlying compute consumption, teams can easily determine if expensive models deliver enough semantic value over faster, lower-cost alternatives. "Generative AI has redefined application health, making evaluation of response quality and model efficiency just as critical as uptime," said New Relic Chief Product Officer Brian Emerson. "Unlike point solutions that evaluate isolated LLM calls, we look holistically across the entire application transaction to show exactly what happens whenever AI is involved, giving teams full visibility into technical health, business impact, and performance - all within the platform tools that SREs, platform engineers, and developers already use." Key features and benefits of the new capabilities include: Real-Time AI Evaluation and Guardrails: * Direct Trace Integration for Rapid Troubleshooting: AI Evaluation attaches probabilistic quality scores to core deterministic distributed traces. Instead of chasing bad answers through siloed logs, engineers can view the prompt payload, judge reasoning, and cross-stack application behaviour in a single screen to instantly isolate and resolve issues. * Automated Guardrail Checks: Configurable guardrails evaluate sampled inputs and outputs to help spot malicious prompt injections, jailbreak attempts, and accidental PII leaks, toxicity, and bias so that engineering teams can take action before these issues cause reputational damage, regulatory fines, or other issues. * RAG Architecture Efficacy: By helping to measure metrics like faithfulness and answer relevancy, engineers can separate a model's reasoning performance from a vector database's retrieval logic to help determine exactly why a response failed to meet its standards. * Connecting AI Quality to Compute Cost: AI Evaluation helps operators identify when models are underperforming qualitatively relative to their token cost, enabling teams to switch to better performing and affordable models without sacrificing the end-user experience. * Out-of-the-box Evaluators: Pre-built evaluators simplify the configuration process for users. Experimental Environment for Rigorous Evaluations Pre-Production: * Prompt Playground: Engineers can test and refine prompts against real models side-by-side in a secure environment before deployment. * Reusable Datasets: Teams can easily curate and version "golden datasets," sourced either from their real distributed traces or synthetic data, to run reliable regression tests on any changes. * Prompt Tracking & Controlled Experiments: Organisations can run controlled A/B testing across different prompts, models, and configurations, reducing the guesswork of prompt engineering. "As organisations move generative AI applications from early pilots into mission-critical production environments, traditional application performance metrics are no longer sufficient on their own," said Stephen Elliot, Group Vice President, I&O, Cloud Operations, and DevOps at IDC. "A successful AI implementation requires visibility into both technical health and response quality, including accuracy, safety, and model efficiency. Bridging live response evaluation with prompt lifecycle management and full-stack operational telemetry is becoming essential for enterprise engineering and security teams looking to mitigate risk and manage costs effectively."