Hot Chips 2026 tutorial explores High Bandwidth Flash as DRAM alternative for ML workloads

High Bandwidth Flash (HBF) is a proposed memory technology that integrates flash storage into the same package as a compute chip, mimicking the form factor of High Bandwidth Memory (HBM). Unlike HBM, HBF prioritizes capacity over bandwidth, targeting machine learning workloads constrained by DRAM limits. The technology does not yet exist in commercial products, so current discussions focus on simulations and software adaptation strategies. HBF’s appeal lies in its potential to offer significantly more memory per dollar than DRAM, but this comes at the cost of reduced bandwidth and increased software complexity.

HBF’s architecture treats flash memory as a block storage device rather than system memory, requiring software to manage data transfers via DMA. This means applications must handle large, aligned accesses, typically 64 KB or larger, rather than byte-level addressing. The approach resembles working with raw disk APIs like `O_DIRECT` in Linux or `FILE_FLAG_NO_BUFFERING` in Windows, where the OS does not abstract away the underlying storage mechanics. For machine learning frameworks, this necessitates rewriting components to handle HBF’s access patterns, such as storing MoE experts or KV caches in flash and moving only active data into faster memory.

The trade-offs between capacity and bandwidth define HBF’s practical limitations. For workloads with smaller models or batch sizes, HBF can reduce costs by providing ample memory without hitting bandwidth constraints. However, as models grow or batch sizes increase, bandwidth demands may outstrip HBF’s capabilities, making HBM or DRAM more cost-effective. Additionally, HBF’s reliance on sequential access patterns complicates its use for sparse or random access workloads, such as top-k KV cache reads, unless software implements workarounds like prefetching into DRAM.

Software frameworks like vLLM would require substantial modifications to leverage HBF effectively. Current optimizations, such as offloading model weights to pinned CPU memory, are incompatible with HBF’s block storage model. Instead, frameworks must implement custom strategies for data placement, movement, and caching, which could introduce overhead and complexity. The effort to adapt software for HBF may rival the work needed to optimize for traditional SSDs, where OS-level buffering can simplify block-aligned access. This raises questions about HBF’s long-term viability compared to incremental improvements in SSD-based solutions.

HBF’s potential to reduce cross-device communication in multi-GPU setups offers another use case. By replicating model weights across GPUs using HBF, frameworks could minimize expensive scatter-gather operations. However, the cost of DMA transfers from flash must be weighed against the performance gains. If HBF’s bandwidth becomes a bottleneck, the cost-per-token equation may shift unfavorably, limiting its applicability to specific scenarios where capacity, not speed, is the primary constraint.