Design and Analysis of L3 Cache Embedded-GPU-High Bandwidth Memory Architecture with Reduced Energy and Latency for AI Computing

For the first time, this paper proposes a L3 cache embedded-GPU-High bandwidth memory (L3E-GPU-HBM) for reduced latency and enhanced energy efficiency of large scale memory intensive AI computing. Accessing HBM in conventional GPU-HBM architecture involves significant latency and requires high data movement energy. To address the challenge, we propose L3E-GPU-HBM in which L3 cache is embedded in interposer between GPU and HBM. To implement the proposed architecture, embedded SRAM interconnect (ESI) chip is employed, which consists of local silicon interconnect (LSI) die and L3 cache die, merged by hybrid bonding. Then, using Chip-on-Wafer-on-Substrate with Local interconnect (CoWoS-L) method, ESI chip is placed inside the reconstituted interposer (RI). The ESI chip functions as both interconnect and L3 cache between L2 cache of GPU and HBM. For verification of the proposed architecture, the circuit model of driver and channel is utilized to obtain the wire latency and energy. The result showed that the proposed L3E-GPU-HBM architecture reduced the wire latency and energy compared to conventional GPU-HBM architecture by 17% and 33 %, respectively.