Forward Deployed Engineer

Maximizing the Power of NVIDIA GB300 NVL72: NVLink Domain-Aware Placement Groups in Ray. Summary. NVIDIA GB300 NVL72 rack-scale systems unlock massive performance boosts by connecting 72 NVIDIA Blackwell GPUs and 36 NVIDIA Grace CPUs together with a rack-scale NVIDIA NVLink scale-up fabric known as an NVLink Domain. To help Ray users take advantage of this topology, Anyscale, Inc. is introducing NVLink Domain-Aware Placement Groups: a multi-node topology-aware scheduling primitive that colocates placement group bundles within a single NVLink Domain. In this blog, Anyscale, Inc. explain why this matters for GB200 and GB300 NVL72 systems and share how NVIDIA's GEAR research lab validated these benefits on real-world NVIDIA GB300 NVL72 training workloads. Until now, Ray placement groups focused on the type of accelerator attached to individual nodes, like choosing between an NVIDIA Hopper or NVIDIA Blackwell GPU. However, newer systems like NVIDIA's GB300 NVL72 provide multi-node NVLink Domains which need topology aware placement to be used effectively. NVLink Domain-Aware Placement Groups extend Ray's scheduling model with topology awareness, allowing users to both maximize high-performance GPU communication and preserve topology intent during disruptions. This small but powerful change unlocks the potential of systems like GB200 and GB300 while laying the groundwork for more general topology-aware scheduling and autoscaling features for multi-host/multi-rack systems. Multi-Node NVLink Scale-up: NVIDIA GB300 NVL72. Traditional GPU servers like the NVIDIA DGX H100 and DGX H200 pack 8 GPUs into a single node connected by NVLink over SXM, with communication between nodes generally using InfiniBand or RoCE. The NVIDIA GB200 NVL72 and GB300 NVL72 provide a new computing platform by connecting a full rack of 72 Blackwell GPUs and 36 Grace CPUs with 1,800 GB/s of all-to-all bandwidth per GPU, effectively enabling all GPUs to work together as one. For workloads where collective communications and memory transfer dominate, the GB200 and GB300 NVL72 are industry leading. Introducing ray NVLink Domain-Aware Placement Groups. To take advantage of NVIDIA GB200 NVL72 and GB300 NVL72 in Ray, users need the ability to request groups of actors be placed within the same NVLink Domain, which corresponds to a rack. Previously Ray placement groups did not understand cross-node dependencies. That is, they could only PACK or STRICT_PACK at the node level. For example, if you tried to STRICT_PACK 18 bundles requiring 4 GPUs and 2 CPUs each in a placement group, it would not be scheduled because you would actually be requesting a single node which has 72 GPUs! The previous workaround was to assign a custom label per GB300 NVL72 rack (e.g., "ray.io/gpu-domain"="rack-7") and then use bundle_label_selector to "pin" bundles to that rack. This is manual and it doesn't compose well with autoscaling or fault-tolerance. To address this, Anyscale, Inc. is introducing NVLink Domain awareness to the placement group API. Users activate this feature by defining a new topology strategy field in the placement group API, i.e. topology_strategy = {ray.io/node-id: "PACK", ray.io/gpu-domain: "STRICT_PACK"}. This new addition tells Ray it MUST place all bundles within one NVLink Domain. For example, in a Ray cluster with multiple GB300 racks, the following placement group would ensure your actors are placed in the same NVLink Domain: 1bundles = [{"GPU": 4, "CPU": 2}] * 16 2topology_strategy = {ray.io/node-id: "PACK", ray.io/gpu-domain: "STRICT_PACK"} 3pg = placement_group( 4bundles=bundles, 5 topology_strategy=topology_strategy, 6 name="gb300_single_rack_16_node_pg", 7) 8ray.get(pg.ready While this code looks simple, the benefits of it are substantial. Optimized networking performance. The most obvious benefit of domain-aware placement is increased networking performance. Keeping tightly coupled actors within the same NVLink Domain allows for collective operations such as all_reduce to use NVLink instead of slower cross-domain paths. Not only that it also allows Anyscale, Inc. to take better advantage of new features to accelerate collective communications such as NVIDIA SHARP[1]. Application-aware fault tolerance. Ray will preserve the application's topology intent when cluster state changes. If a node fails or requires maintenance, Ray will replace the node from the same NVLink Domain when capacity exists rather than silently fragmenting the workload across racks. An example of this can be seen in Figure 3. If capacity does not exist, the bundle is queued for rescheduling unless the entire set of bundles goes down where a new NVLink Domain is automatically selected for use. This fault tolerance mechanism has always been an important part of Ray placement groups and now it is extended to these multi-node systems. Real world example: large scale distributed VLA training on GB300. Anyscale, Inc. partnered with NVIDIA's GEAR research lab to test this new feature across their large scale VLA (Vision Language Action) pre-training runs on a GB300 cluster. To evaluate the benefit of NVLink Domain-Aware Placement Groups in Ray, NVIDIA GEAR undertook two training runs, one with, and one without this new feature. Each run used 512 GPUs (128 nodes), composed of 8 groups of 64 GPUs (16 nodes). The second run achieved 1.13x faster iterations per second than the first. However, this is just the beginning, as further opportunities remain to make even more effective use of the GB300's unique hardware capabilities. Why is the performance better in run two? In run one, GPUs spend more time exchanging data with their placement group peers (for example during collective operations like all_reduce). Whereas in run two, actors are placed such that they benefit from NVLink's fast GPU to GPU memory sharing which enables training speeds not possible in the first run. Furthermore, by making Ray aware that placement groups are intended to be located on the same NVLink Domain, handling faults gracefully during long training runs is significantly easier. Other use-cases for rack-aware scheduling. Distributed inference. Serving workloads that use disaggregated inference often separate prefill and decode workers. On GB300-class systems, it is especially important that these workers land within the same NVLink Domain so they can exchange KV cache and intermediate state over the fastest available path. NVLink Domain-Aware Placement Groups make this much easier allowing users to express the intent that related prefill and decode actors should be colocated on the same rack, instead of manually pinning them with rack-specific labels. Reinforcement learning. RL workloads have many phases: training, rollout, and evaluation, each with different communication patterns. Domain-aware placement lets users keep the most tightly coupled parts of the pipeline together, such as policy training workers or actors that frequently exchange model weights, while still giving Ray flexibility to schedule less communication-heavy components elsewhere. What's next? The primary goal of this feature is to bring awareness of NVLink Domains to the Ray application layer. Over the next few releases, Anyscale, Inc. aim to provide even more visibility and tools to understand GPU locality and optimize placement for maximum performance and interconnect efficiency. Anyscale, Inc.'ll also be expanding the Placement Group API on several fronts: * NVLink Domain-Aware Placement Groups currently only support STRICT_PACK. Anyscale, Inc. plan to add support for other strategies like STRICT_SPREAD, which would, for example, let you place bundles on separate racks. * Nested topologies: The topology expressed via NVLink Domain-Aware Placement Groups is flat today. In reality, infrastructure is usually hierarchical where racks live within datacenters, which live within availability zones, and so on. Anyscale, Inc. plan to add support for expressing these nested topologies. Try it out now. Anyscale, Inc.'d love to hear feedback from the community on this feature. If you have ideas for improving the API or run into issues using it, please open a github issue on https://github.com/ray-project/ray. Table of contents.

Cloud provider Nscale has agreed to acquire software startup Anyscale for approximately $1.65 billion. San Francisco-based Anyscale offers software that helps artificial intelligence workloads run more effectively across separate servers and data centres. Formed from a cryptocurrency mining business in early 2024, London-based Nscale leases AI-focused computing power and is developing massive data centres in Norway and West Virginia. The acquisition will allow Nscale customers to access training, fine-tuning, and inference services through a single platform. Anyscale reported 70% revenue growth in its most recent quarter. All roughly 200 employees will join Nscale. The company previously raised funds in 2022 above a $1 billion valuation. Nscale plans to IPO, potentially in the second half of 2026.