Humans& ships Weight Teleportation to sync 445 GB AI models in 220 ms. humans& open-sourced a lossless GPU compression pipeline that cuts RL sampler pause time to 0.23s, beating intra-cluster RDMA for weight sync. · 3 hrs ago Read 7 min * humans& released GPU-accelerated Weight Teleportation, lossless RL weight sync faster than intra-cluster RDMA. * Sampler pause drops to 219.6 ms; end-to-end sync under 4 s on 445 GB NVFP4 GLM-5.3. * Snappy plus Zstd double compression shrinks per-step delta payload to 2.65 GB, 0.60% of model weights. * Heterogeneous pipeline: CPU handles outer Zstd, Blackwell DE ASIC streams Snappy from pinned host memory into HBM. * Double-buffered DE-SM overlap means no KV cache eviction and Triton XOR kernels hide behind decompression. * Open-sourced in SGLang and Miles with a one-shot update API. Humans& has published Weight Teleportation, an open-source pipeline for moving updated model weights from reinforcement-learning trainers to rollout samplers. In its reference benchmark, the system compressed a 445 GB model update to 2.65 GB, completed end-to-end synchronization in under four seconds, and paused sampling for 219.6 ms. The design combines bitwise deltas, nested lossless compression, CPU staging, Blackwell's fixed-function Decompression Engine, and Triton kernels. Its measured application speed reaches the decompression hardware's throughput limit. Why weight movement stalls rollout. Large-scale RL systems use trainers to update a policy and samplers to run that policy while generating trajectories. Those samplers need regular weight updates because growing version lag changes the policy that produced the training data, while blocking updates leave inference GPUs idle. Full checkpoints can span hundreds of gigabytes. Sending them frequently requires substantial network capacity, and loading them can evict the key-value cache used for inference. Weight Teleportation reduces the transferred bytes and performs most receiver work while rollout decoding continues. The benchmark behind the claim. The reference workload uses an NVFP4 GLM-5.3 checkpoint containing 445 GB of canonical weight tensors. The trainer side spans eight nodes with 64 B300 GPUs, and each simulated step mutates 0.085% of the checkpoint as a proxy for a steady-state RL update. * Canonical weight size: 445 GB * Trainer hardware: 64 B300 GPUs across eight nodes * Mutation rate: 0.085% per step * Compressed payload: 2.65 GB, about 0.60% of the checkpoint * End-to-end synchronization: under four seconds * Sampler pause: 219.6 ms The mutation rate is central to the result. Denser or less compressible updates would increase payload size, network time, staging requirements, and potentially the sampler pause. Bitwise deltas expose the zeros. The trainer treats the old and new weights as flat bit buffers and computes their exclusive OR, or XOR. Each zero marks an unchanged bit, while each one marks a flipped bit. The receiver applies the same XOR operation to its old weights and reconstructs the new weights exactly. * The trainer XORs the previous and current weight buffers. * Lossless codecs compress the resulting delta. * The receiver decompresses the delta into temporary layer buffers. * A Triton kernel XORs each buffer into the live weights. A raw XOR mask occupies as many bytes as the original weights. Its compression ratio comes from repeated zeros when relatively few bits change. Because reconstruction uses bitwise operations, the method preserves every bit and works across floating-point and low-precision formats such as NVFP4. Both endpoints must share the same base weights, byte layout, and version. A missing or misidentified update would produce the wrong parameters, so production integrations need strict checkpoint and delta version tracking. Two codecs produce the useful stream. A direct GPU codec leaves too many bytes on the wire in this workload. GPU LZ4 produced a 5.36 GB payload, 6.1 times the 0.876 GB generated by a CPU Zstd baseline. Fully decoding the tensor delta on the CPU would sacrifice the direct hardware path into GPU memory, so the pipeline nests two codecs. Reported compression results for the 445 GB delta buffer | Codec path | Inner payload | After Zstd | | Snappy, then Zstd | 19.78 GB | 2.65 GB | | LZ4, then Zstd | 5.36 GB | 4.03 GB | Snappy creates the larger first-stage payload, but its simpler copy commands give Zstd more repetitive structure to compress. Zstd reduces the Snappy stream to about 13% of its original size, compared with roughly 75% for the LZ4 stream. Blackwell turns decoding into a stream. The receiver decodes the outer Zstd layer on the CPU while rollout inference continues. It places the resulting Snappy stream in pinned host memory, then uses Blackwell's Decompression Engine to read that stream directly and write decompressed layer data into high-bandwidth GPU memory. CUDA exposes this path through the driver allocation flag CU_MEM_CREATE_USAGE_HW_DECOMPRESS. The Decompression Engine is a fixed-function block separate from the GPU's streaming multiprocessors, allowing hardware decompression to overlap with other compute. A double-buffered pipeline keeps both engines occupied. The Decompression Engine writes one layer into buffer A while a Triton kernel applies the delta from buffer B to the live weights, after which the buffers swap roles. Only two layers need temporary GPU storage, so the implementation can use memory already reserved for inference activations and preserve the key-value cache. The Triton XOR work fits behind decompression latency in the reported workload. Once that overlap is complete, the Decompression Engine sets the lower bound for the blocking application phase. Measured costs. Author-reported comparison with a GPU LZ4 baseline | Stage | GPU LZ4 baseline | Weight Teleportation | | Trainer preparation and compression | 0.834 s | 0.789 s | | Receiver work | 6.041 s | 2.355 s, overlapped | | Sampler pause | 6.041 s | 219.6 ms | | Payload size | 5.36 GB | 2.65 GB | Most of the 2.355 seconds of receiver work runs without blocking rollout because the outer decode uses CPU resources and pinned host memory. The sampler pauses during the final hardware-decompression and XOR streaming phase. One delta system covers several workflows. Version-specific deltas let the same infrastructure handle synchronization, recovery, evaluation, and deployment without separate checkpoint-transfer mechanisms. * Evaluation workers: An evaluator several steps behind receives a delta computed against its last loaded version. * Fault recovery: A restarted rollout or evaluation worker loads its base checkpoint and requests a delta from that exact version. * RL startup: A run resumed from a trained checkpoint uses LZ4 plus Zstd for the larger accumulated delta. The smaller inner LZ4 stream limits receiver staging memory. * Remote deployment: A serving process starts from a shared base checkpoint and requests current weights through /update_weights_from_gpu_delta. The authors also present full-parameter deltas as an option for serving workloads that might otherwise use LoRA adapters. The delta format can represent arbitrary weight changes without imposing a rank limit, while its size depends on bit-level compressibility and access to a shared base checkpoint. Code lands in Miles and SGLang. Miles contains orchestration, distributed GPU compression, fault tolerance, and delta checkpointing. SGLang contains the heterogeneous receiver pipeline, Decompression Engine overlap, Triton kernels, and one-shot weight-update API. The accelerated receiver path targets Blackwell hardware, including B200 and B300 systems, and depends on CUDA driver support for hardware-decompression allocations. Teams using earlier GPU generations would need another decode path and should expect different pause times. Where the benchmark may bend. * Update density: The reported workload mutates 0.085% of the checkpoint. Compression will weaken as more bits change. * Shared state: Sender and receiver need identical base weights, tensor layouts, and version metadata. * Host resources: The receiver needs pinned CPU memory for the inner compressed stream and enough CPU throughput for Zstd. * Hardware: The shortest blocking path relies on Blackwell's Decompression Engine and its supported codecs. * Network: End-to-end time still depends on bandwidth, latency, topology, and concurrent traffic. * Validation: The published figures come from the project team's proxy workload and await independent reproduction on complete RL training runs. Humans& reports that the compressed pipeline can outperform intra-cluster peer-to-peer RDMA for many practical workloads because it sends fewer bytes and overlaps receiver work. Cross-datacenter sampling becomes feasible when the compressed payload, available bandwidth, and permitted policy lag fit the training loop's requirements. The benchmark shows how a sparse, exact weight delta can become a hardware-decodable stream whose final application takes about 220 ms. For Blackwell-based RL systems, the available code provides a concrete integration path and a measurable baseline for evaluating update density, network cost, host-memory pressure, and sampler downtime.

What is the trickle test? Humans& trained a model to act like a person instead of a helpful assistant, and their evals for it - group chats, and a "trickle test" for whether a model reveals facts at a human pace - are worth a look. Humans&, a "human centered AI lab," launched an interesting model that is trained to not act like "a helpful assistant," but the way humans do. More messy, not always as helpful, and not as easily identifiable as a model. I have a lot of thoughts about this, but reading through their post, I found it particularly interesting (I do teach evals after all) to see how they set up their evals to train the model to be more "human like." (Again, which is often not necessarily a good thing.) They figured out that group conversations is a useful way to train model to be more human. There's a lot of data (Slack chats etc.), and humans behave differently than the current LLMs do. They're not as helpful. They provide less context. Here's a scenario they used: Three university students have a group presentation due in four days. Two have drafted their sections. The third has missed a planning call and uploaded only an outline. Their shared grade includes a peer-contribution assessment. They are meeting in a chat to decide how to finish The whole thing made me think about the "synthetic user" debate as well. My feelings on that are complicated and I don't have a strong take, except for: * You can't just ask an LLM "what your users would do" etc., that misunderstands fundamentally what user research is. * At the same time, you can indeed generate synthetic user activity if it's based on real user activity to test your systems, for example. That can work. You need to be careful and smart about it though. Another interesting eval is their "Trickle Test." Basically: LLMs tend to give all the context you need in one answer, whereas humans will share a few things and then share a few more bits and pieces and then get distracted etc. So the test checks whether the model reveals facts at the same pace real people do, rather than dumping everything upfront. It takes a real human conversation where certain facts were known to that person, feed those facts to the model as "ground truth," and have the model replay the same conversation turn by turn. At each turn, compare what the model has revealed against what the human had actually revealed by that point. I just find it all very interesting, and I keep expanding my intuition around evals, my "eval brain." It kind of reminds me of when I learned to do object modeling. First it broke my brain, and then I saw everything in terms of models (and I still do). Evals is similar. Health and happiness, Peter