AI and Memory Wall

The availability of unprecedented unsupervised training data, along withneural scaling laws, has resulted in an unprecedented surge in model size andcompute requirements for serving/training LLMs. However, the main performancebottleneck is increasingly shifting to memory bandwidth. Over the past 20years, peak server hardware FLOPS has been scaling at 3.0x/2yrs, outpacing thegrowth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and1.4 times every 2 years, respectively. This disparity has made memory, ratherthan compute, the primary bottleneck in AI applications, particularly inserving. Here, we analyze encoder and decoder Transformer models and show howmemory bandwidth can become the dominant bottleneck for decoder models. Weargue for a redesign in model architecture, training, and deployment strategiesto overcome this memory limitation.