Head of Customer Engineering

Maximizing the Power of NVIDIA GB300 NVL72: NVLink Domain-Aware Placement Groups in Ray. Summary. NVIDIA GB300 NVL72 rack-scale systems unlock massive performance boosts by connecting 72 NVIDIA Blackwell GPUs and 36 NVIDIA Grace CPUs together with a rack-scale NVIDIA NVLink scale-up fabric known as an NVLink Domain. To help Ray users take advantage of this topology, Anyscale, Inc. is introducing NVLink Domain-Aware Placement Groups: a multi-node topology-aware scheduling primitive that colocates placement group bundles within a single NVLink Domain. In this blog, Anyscale, Inc. explain why this matters for GB200 and GB300 NVL72 systems and share how NVIDIA's GEAR research lab validated these benefits on real-world NVIDIA GB300 NVL72 training workloads. Until now, Ray placement groups focused on the type of accelerator attached to individual nodes, like choosing between an NVIDIA Hopper or NVIDIA Blackwell GPU. However, newer systems like NVIDIA's GB300 NVL72 provide multi-node NVLink Domains which need topology aware placement to be used effectively. NVLink Domain-Aware Placement Groups extend Ray's scheduling model with topology awareness, allowing users to both maximize high-performance GPU communication and preserve topology intent during disruptions. This small but powerful change unlocks the potential of systems like GB200 and GB300 while laying the groundwork for more general topology-aware scheduling and autoscaling features for multi-host/multi-rack systems. Multi-Node NVLink Scale-up: NVIDIA GB300 NVL72. Traditional GPU servers like the NVIDIA DGX H100 and DGX H200 pack 8 GPUs into a single node connected by NVLink over SXM, with communication between nodes generally using InfiniBand or RoCE. The NVIDIA GB200 NVL72 and GB300 NVL72 provide a new computing platform by connecting a full rack of 72 Blackwell GPUs and 36 Grace CPUs with 1,800 GB/s of all-to-all bandwidth per GPU, effectively enabling all GPUs to work together as one. For workloads where collective communications and memory transfer dominate, the GB200 and GB300 NVL72 are industry leading. Introducing ray NVLink Domain-Aware Placement Groups. To take advantage of NVIDIA GB200 NVL72 and GB300 NVL72 in Ray, users need the ability to request groups of actors be placed within the same NVLink Domain, which corresponds to a rack. Previously Ray placement groups did not understand cross-node dependencies. That is, they could only PACK or STRICT_PACK at the node level. For example, if you tried to STRICT_PACK 18 bundles requiring 4 GPUs and 2 CPUs each in a placement group, it would not be scheduled because you would actually be requesting a single node which has 72 GPUs! The previous workaround was to assign a custom label per GB300 NVL72 rack (e.g., "ray.io/gpu-domain"="rack-7") and then use bundle_label_selector to "pin" bundles to that rack. This is manual and it doesn't compose well with autoscaling or fault-tolerance. To address this, Anyscale, Inc. is introducing NVLink Domain awareness to the placement group API. Users activate this feature by defining a new topology strategy field in the placement group API, i.e. topology_strategy = {ray.io/node-id: "PACK", ray.io/gpu-domain: "STRICT_PACK"}. This new addition tells Ray it MUST place all bundles within one NVLink Domain. For example, in a Ray cluster with multiple GB300 racks, the following placement group would ensure your actors are placed in the same NVLink Domain: 1bundles = [{"GPU": 4, "CPU": 2}] * 16 2topology_strategy = {ray.io/node-id: "PACK", ray.io/gpu-domain: "STRICT_PACK"} 3pg = placement_group( 4bundles=bundles, 5 topology_strategy=topology_strategy, 6 name="gb300_single_rack_16_node_pg", 7) 8ray.get(pg.ready While this code looks simple, the benefits of it are substantial. Optimized networking performance. The most obvious benefit of domain-aware placement is increased networking performance. Keeping tightly coupled actors within the same NVLink Domain allows for collective operations such as all_reduce to use NVLink instead of slower cross-domain paths. Not only that it also allows Anyscale, Inc. to take better advantage of new features to accelerate collective communications such as NVIDIA SHARP[1]. Application-aware fault tolerance. Ray will preserve the application's topology intent when cluster state changes. If a node fails or requires maintenance, Ray will replace the node from the same NVLink Domain when capacity exists rather than silently fragmenting the workload across racks. An example of this can be seen in Figure 3. If capacity does not exist, the bundle is queued for rescheduling unless the entire set of bundles goes down where a new NVLink Domain is automatically selected for use. This fault tolerance mechanism has always been an important part of Ray placement groups and now it is extended to these multi-node systems. Real world example: large scale distributed VLA training on GB300. Anyscale, Inc. partnered with NVIDIA's GEAR research lab to test this new feature across their large scale VLA (Vision Language Action) pre-training runs on a GB300 cluster. To evaluate the benefit of NVLink Domain-Aware Placement Groups in Ray, NVIDIA GEAR undertook two training runs, one with, and one without this new feature. Each run used 512 GPUs (128 nodes), composed of 8 groups of 64 GPUs (16 nodes). The second run achieved 1.13x faster iterations per second than the first. However, this is just the beginning, as further opportunities remain to make even more effective use of the GB300's unique hardware capabilities. Why is the performance better in run two? In run one, GPUs spend more time exchanging data with their placement group peers (for example during collective operations like all_reduce). Whereas in run two, actors are placed such that they benefit from NVLink's fast GPU to GPU memory sharing which enables training speeds not possible in the first run. Furthermore, by making Ray aware that placement groups are intended to be located on the same NVLink Domain, handling faults gracefully during long training runs is significantly easier. Other use-cases for rack-aware scheduling. Distributed inference. Serving workloads that use disaggregated inference often separate prefill and decode workers. On GB300-class systems, it is especially important that these workers land within the same NVLink Domain so they can exchange KV cache and intermediate state over the fastest available path. NVLink Domain-Aware Placement Groups make this much easier allowing users to express the intent that related prefill and decode actors should be colocated on the same rack, instead of manually pinning them with rack-specific labels. Reinforcement learning. RL workloads have many phases: training, rollout, and evaluation, each with different communication patterns. Domain-aware placement lets users keep the most tightly coupled parts of the pipeline together, such as policy training workers or actors that frequently exchange model weights, while still giving Ray flexibility to schedule less communication-heavy components elsewhere. What's next? The primary goal of this feature is to bring awareness of NVLink Domains to the Ray application layer. Over the next few releases, Anyscale, Inc. aim to provide even more visibility and tools to understand GPU locality and optimize placement for maximum performance and interconnect efficiency. Anyscale, Inc.'ll also be expanding the Placement Group API on several fronts: * NVLink Domain-Aware Placement Groups currently only support STRICT_PACK. Anyscale, Inc. plan to add support for other strategies like STRICT_SPREAD, which would, for example, let you place bundles on separate racks. * Nested topologies: The topology expressed via NVLink Domain-Aware Placement Groups is flat today. In reality, infrastructure is usually hierarchical where racks live within datacenters, which live within availability zones, and so on. Anyscale, Inc. plan to add support for expressing these nested topologies. Try it out now. Anyscale, Inc.'d love to hear feedback from the community on this feature. If you have ideas for improving the API or run into issues using it, please open a github issue on https://github.com/ray-project/ray. Table of contents.

Cloud provider Nscale has agreed to acquire software startup Anyscale for approximately $1.65 billion. San Francisco-based Anyscale offers software that helps artificial intelligence workloads run more effectively across separate servers and data centres. Formed from a cryptocurrency mining business in early 2024, London-based Nscale leases AI-focused computing power and is developing massive data centres in Norway and West Virginia. The acquisition will allow Nscale customers to access training, fine-tuning, and inference services through a single platform. Anyscale reported 70% revenue growth in its most recent quarter. All roughly 200 employees will join Nscale. The company previously raised funds in 2022 above a $1 billion valuation. Nscale plans to IPO, potentially in the second half of 2026.

Anyscale signs definitive agreement to join Nscale. What this means: * Doubling down on Ray. Anyscale, Inc. is expanding its investment in Ray and the open-source community. Together, Anyscale, Inc. will directly optimize Ray for cutting-edge accelerator and data center architectures. * An open source strategy. Under PyTorch Foundation governance, contributions from Google, NVIDIA, Microsoft, and the community are growing. Nscale plans to join the Foundation as a platinum member. Open-source is at the heart of Nscale's strategy, as it is ours. * More GPU capacity. Anyscale Platform customers will gain access to significant compute capacity from Nscale. * Multi-cloud flexibility. Post-closing, the Anyscale Platform will continue to run across all major cloud providers. Portability remains core to its roadmap for both Ray and the Anyscale Platform. The bottleneck now spans the stack. When Anyscale, Inc. created Ray at UC Berkeley and launched Anyscale, Anyscale, Inc. believed AI compute needs would explode. The first bottleneck was the software for distributed computing, and Anyscale, Inc. built Ray to address it. That bet played out. Ray is now used across every major AI workload, from data preparation to training to inference, and is used to build many frontier model families, including GLM, Nemotron, Composer, and MAI. But AI systems have grown orders of magnitude in scale and complexity. Data processing is becoming multimodal, inference-heavy, and GPU-based. Reinforcement learning mixes training, inference, and simulation together in a single workload. Inference requires disaggregation, GPU memory management for extremely long context, and complex routing and failure handling for mixture-of-experts architectures. These challenges are inseparable from the hardware. Software must account for rack and cluster topology, capacity, hardware heterogeneity, compute disaggregation, and failures at every level and in every component. Optimizing one layer at a time is no longer enough. The future requires deep, joint optimization across every layer of the software and hardware stack. Why Nscale. Among the neoclouds, Nscale stands out for its execution speed and vision of complete vertical integration. Nscale focuses on more of the physical infrastructure, from land and power to data centers and accelerated compute. Its multi-gigawatt pipeline addresses one of AI's biggest constraints and gives Anyscale, Inc. compute availability and density as well as a tighter feedback loop for joint optimization. In addition, Nscale was among the first to deploy next-generation GB300 NVL72 systems at scale. Beyond physical infrastructure, Nscale has built performant platform software for large-scale inference, immediately accelerating its combined roadmap. Similar to Anyscale, Nscale is betting on open source as a strategy and believes that the winning AI infrastructure standards will be open. They plan to join the PyTorch Foundation as a Platinum member and to invest heavily in the open source ecosystem. Together, Anyscale and Nscale can co-design the software layer and infrastructure beneath it, something that neither company could do as effectively by optimizing its layer alone. Commitment to open source & multi-cloud. Ray was built from day one as an open, community-driven project, and it is governed by the PyTorch Foundation alongside PyTorch and vLLM. Its value as an industry standard depends on it being fully open, neutral, and portable. That openness is why companies across the industry invest in Ray. Over the past year, engineers from Google, NVIDIA, Microsoft, Red Hat, Alibaba, along with the broader Ray community, have improved latest-generation GPU and TPU support, topology-aware scheduling, GPU-native data processing, Kubernetes integration, and the Ray History Server. Going forward, the Anyscale + Nscale team will invest heavily in maintaining and improving Ray. Anyscale, Inc. will continue to bolster the community, mentor contributors, and seek to expand project governance. Ray has a history of co-evolution with other components of the open AI infrastructure stack. Anyscale, Inc. is now extending that co-design deeper into the hardware layer. Portability is a requirement. Ray was designed to support any hardware accelerator, integrate with any ML framework, and run in any environment, including your laptop, on premises, and any cloud provider. That philosophy remains unchanged across Ray and the Anyscale Platform. Looking ahead. This is a critical period of growth, and this past quarter was its strongest yet, with over 70% quarter-over-quarter revenue growth. Anyscale, Inc. is just getting started. Together with Nscale, Anyscale, Inc. will make distributed AI infrastructure simpler, more reliable, and more efficient, all while doubling down on the openness and portability that have made Ray a foundational part of the AI ecosystem. Anyscale, Inc. will share more at Ray Summit in San Francisco this August. Post-closing, Anyscale, Inc. will also be hiring across the combined team! Table of contents.