Micron mentioned it’s now delivery “manufacturing succesful?” HBM3E in 12-Excessive stacks. Which means we get 36GB per package deal, up from 24GB, and better efficiency. For the AI trade, it is a massive deal as a result of it usually means we are able to get higher-performance and higher-capacity accelerators.
Micron HBM3E 12-Excessive 36GB Increased Capability AI Accelerators
The brand new 12-Excessive stack is what it appears like. 12 chips stacked with By way of-Silicon Vias (TSVs) right into a HBM3E dice. This dice is often packaged subsequent to a GPU with a high-performance interconnect by a substrate.
A bonus of getting a 12-high stack versus an 8-high stack is that we are able to enhance the reminiscence capability by 50% from 24GB to 36GB. Whereas that will sound like a no brainer for AI inference and coaching duties which are reminiscence capability and bandwidth-bound, it isn’t so simple as simply stacking extra chips. Not solely to the info and energy connections must be made, however the precise chip itself has Z-axis constraints. For instance, an accelerator just like the AMD MI300X is made up of many items of silicon. You possibly can see the eight HBM stacks on the skin.
If AMD needed to make use of Micron’s HBM3E in one thing like a MI325X, then it must make sure the packages stayed on the similar peak. If this didn’t occur, then the package deal would must be adjusted or there could be an actual threat of cracking GPUs like we noticed occur moderately usually within the NVIDIA P100/ V100 period. Observe AMD has not confirmed Micron as a provider, so we’re simply utilizing this as an illustration. Nonetheless, the advantages are palpable since now we have 36GB HBM3E packages x 8 websites = 288GB of reminiscence.
Along with this, HBM3E additionally presents efficiency advantages over previous-generation elements with 9.2Gbps pin pace and what Micron claims is 1.2TB/s of reminiscence bandwidth.
Last Phrases
36GB HBM3E goes to be an vital product, however one we solely anticipate in higher-end chips. Normally, the large AI distributors like NVIDIA and, to some extent, AMD will get entry to the brand new chips and greatest bins of the chips as a result of they’ll drive quantity and pay essentially the most for them. That can be why we see chips just like the Microsoft MAIA 100 AI Accelerator for Azure use HBM2E as a result of that tends to be decrease price. Hopefully, we begin to see AI accelerators with this quickly. Tomorrow’s overview won’t use these, though will probably be of a cool AI system.