Applied AI Engineer Intern

Hot Chips 2026: d-Matrix stacks AI accelerator directly on custom DRAM for 100 TB/s per card - TSMC 4nm compute die bonded face-to-face at a 36-micron pitch on top of a custom-designed die. Published 3 hours ago Heading into production during the tightest DRAM market in more than a decade. This Tom's Hardware Premium article is free to read with a Tom's Hardware account; no payment necessary. We're offering free access from August 23 to 26 so you can read all of our reporting from Hot Chips. d-Matrix presented Raptor, which it calls the first 3D DRAM accelerator for generative inference, at Hot Chips 2026 this week, showing a TSMC 4nm compute die bonded face-to-face at a 36-micron pitch on top of a custom-designed DRAM die that delivers 100 TB/s of bandwidth from 32GB per card. Co-founder and CTO Sudeep Bhoja put the vertical interface's energy cost at 0.37 pJ/bit against roughly 2.4 pJ/bit for moving data into an HBM4 base die, calling it "a measured number" from working silicon, and the accompanying ISCA 2026 paper, written with the University of British Columbia, projects around 4.7 times higher throughput per card than HBM-based designs. Bhoja, however, didn't disclose who manufactures the DRAM die. Latest Videos FromTom's Hardware How to check SSD health in Windows 10 and Windows 11 0 seconds of 1 minute, 25 seconds Volume 0% CEO Sid Sheth told CNBC in June that Raptor is slated to launch in 2027, but at Hot Chips the company gave no firm date or information on volume and pricing, and every performance figure shown, including 988 tokens per second per user on the 2.8-trillion-parameter Kimi K3 model at 1M-token context, is a d-Matrix projection built on early silicon. The custom DRAM die. Raptor inverts the usual 3D stacking arrangement by putting the logic die on top and the DRAM underneath, so a cold plate sits directly on the compute silicon and the DRAM die doubles as the interposer, carrying PCIe and die-to-die signals down through its TSVs. "For the same amount of power, you can drive the bandwidth up," Bhoja said during the session, "and so we were able to drive the bandwidth up here to 100 terabytes per second." The one-high stack achieves a power density of roughly 0.5W per square millimeter, which liquid cooling can handle, but the DRAM is designed for a junction temperature of 105°C, where retention collapses from a standard 32ms to 4ms, and the memory must refresh eight times more often. d-Matrix absorbed that penalty by shrinking each microbank to 1,366 rows and about 5.33MB, so a full refresh sweep costs only 1.37% of overall bandwidth. The die carries 840 banks per chiplet, of which 72 (around 9%) are spares wired into a two-level mux chain the company calls bank chaining, letting any two failed banks anywhere on the die be switched out while channels stay symmetric. A [132,128] Reed-Solomon code on the logic die corrects two symbol errors per 128 bytes, with a CRC behind it. The interface has no PHY, no burst structure, and no sideband pins, so conventional data-bus inversion was impossible; d-Matrix instead compares each 128-byte flit to the previous one and stores a 1-bit inversion tag alongside the ECC metadata, recovering roughly 20% of the I/O power DBI would have saved. At full tilt, the vertical interface still burns 296W of the 422W per-package budget the ISCA paper discloses. The bank geometry, spare-bank mux tree, refresh behavior, and interleaved ECC columns were all co-designed with the compute die. The 0.37 pJ/bit figure exists only because of that pairing, and no memory maker has anything like this die in its catalog. Who's supplying it? d-Matrix has named TSMC for the N4P logic die and Alchip as its ASIC design and 2.5D/3D packaging partner, but neither company operates a DRAM fab, and across the Hot Chips talk, the ISCA paper, and every public announcement since the Pavehawk 3DIMC test silicon came online last September, the firm has never identified who fabricates its custom DRAM. Only three companies make leading-edge DRAM at volume, and all three are allocating capacity to HBM4 lines that are effectively sold out through 2026. J.P. Morgan estimates DRAM prices will have risen more than 400% between the start of 2024 and the end of 2026. In addition, analysts have recorded contract price increases of 90% to 95% in Q1 2026 alone, and SK hynix CEO Kwak Noh-jung told Reuters in July that "customer demand will remain higher than our supply capacity even beyond 2030." Nvidia, the memory makers' largest and most leveraged customer, is reportedly testing Rubin Ultra configurations with as little as 192GB because it may not be able to source enough HBM4E. A startup with roughly $450 million raised, asking a memory maker to run a bespoke die with non-standard bank geometry on capacity that could otherwise print HBM, is negotiating from a far weaker position than that, and until the supplier is named, Raptor's 2027 volume plan rests entirely on an unknown, undisclosed dependency. The die's 11.4 MB/mm[2] density is roughly half of HBM4's 21.9 to 26.3 MB/mm[2], Bhoja acknowledged during the Q&A: "A lot of the drop for us was also because we used a not-so-advanced DRAM. And so if we used a more mainline DRAM, just like the HBM4 guys are doing, we would be able to push that up almost all the way to the HBM4 numbers." The density penalty therefore tracks back to whatever foundry arrangement d-Matrix currently has. 32GB per card. Raptor's 32GB per card stands against 192GB to 288GB for HBM4-equipped accelerators, so d-Matrix sizes deployments at rack scale instead: 72 cards carry 2.3TB, enough to hold Kimi K3's weights at 4-bit precision with headroom for around 54 concurrent users at 1M context, by its own calculations. "Even with 32 gigabytes of memory capacity, we are able to solve SOTA models in a scale-up network, so no bits are wasted," Bhoja said. KV cache growth works against that arithmetic over time, and co-presenter Aayush Ankit, who led Raptor's SoC architecture at d-Matrix before joining Meta's MTIA team, explained the failure mode while dismissing SRAM alternatives: "We are making this unit of compute blazingly fast. Communication becomes a bottleneck soon enough." Once models and context no longer fit a single rack, inference spills into the inter-card synchronization overhead that the vertical bandwidth was meant to eliminate, which the ISCA paper concedes. Cerebras claimed 969 tokens per second on Llama 3.1 405B in November 2024, with third-party firm Artificial Analysis verifying the figure on live hardware, and that remains the closest published reference point for Raptor's numbers. d-Matrix's 988 tokens per second per user on a model seven times larger would be a step change if it holds water, but no third party has measured Raptor, and the comparison points in the ISCA paper are simulations anchored to early silicon characterization. Asked by an Nvidia employee about multi-layer stacking plans, Bhoja said: "Our roadmap is still a work in progress, and we have a hard enough time trying to get one-high to work and work around all of the thermal issues of that." Samsung brought its own version of the idea to Hot Chips with zHBM, a concept that stacks HBM directly on the processor and carries a claimed 70% power-efficiency gain over an HBM4E setup, with no production timeline before HBM5. The problem for d-Matrix is that Samsung runs its own DRAM fabs; d-Matrix doesn't. Whether it can get wafers at volume remains to be seen. Full d-Matrix Hot Chips 2026 presentation. Image 1 of 30 Gain access to Tom's Hardware Premium's Hot Chips 2026 coverage between 23-26 Aug with a free account * | Get deeper news analysis * | Access detailed hardware roadmaps * | Explore performance data with our exclusive Bench tool Stay signed in for an improved experience across Tom's Hardware Contributor Luke James is a freelance writer and journalist. Although his background is in legal, he has a personal interest in all things tech, especially hardware and microelectronics, and anything regulatory.

d-Matrix, a Santa Clara-based AI chip company, raised $275 million in Series C funding at a $2 billion valuation, bringing total funding to $450 million. The round was led by Bullhound Capital, Triatomic Capital, and Temasek, with new investors including Qatar Investment Authority and EDBI. The company will use the funds to advance its product roadmap, expand globally, and support deployments of its data centre inference platform. d-Matrix's products include Corsair inference accelerators, JetStream networking accelerators, and Aviator software. The platform claims to produce up to 30,000 tokens per second at 2 milliseconds per token on a Llama 70B model and can run models with up to 100 billion parameters in a single rack.