A Survey of Compute-in-Memory Technologies for AI Acceleration

As an engineer who spent years optimizing software bottlenecks, Jianshi Tang's survey of compute-in-memory (CIM) chips hits right at the core of our current hardware limits. We are living through an AI boom where training and inference are incredibly power-hungry, largely due to the von Neumann bottleneck—the constant shuttling of data between separate processing units and memory chips. Tang's review of emerging non-volatile memory devices, like ReRAM and PCM, highlights a paradigm shift where we perform computations directly inside the memory array itself. This radically reduces latency and power consumption, which is exactly the kind of efficiency leap needed to make edge AI viable.

From a startup founder's view, CIM is a glimpse into the next wave of infrastructure disruption. It is easy to assume that scaling AI just means buying more GPU clusters, but the real winners of the next decade will likely build on hardware that breaks the power-performance trade-off. Seeing how far non-volatile memory technology has come makes me think about how we should prepare our software stacks for a world where compute is completely decentralized down to the memory cell. It is a reminder that while high-level software models grab all the headlines, the ultimate boundary of what we can build is always defined by the physics of the silicon beneath it.

What stuck with me

• Von Neumann bottleneck: Traditional computing architectures waste massive amounts of energy simply moving data between processor and memory rather than doing actual computation.
• Emerging memory devices: Technologies like resistive RAM and phase-change memory are enabling practical, highly efficient implementations of analog compute-in-memory chips.
• Edge AI enablement: Minimizing power consumption through compute-in-memory architecture is the key to running complex artificial intelligence models locally on resource-constrained devices.