Cache or Scratchpad? Why Choose?

The following is a special contribution to this blog by CCC Executive Council Member Mark D. Hill of the University of Wisconsin-Madison. Full disclosure: He had the pleasure of working with one of the authors of the discussed paper—Sarita Adve—on her 1993 Ph.D.

Great conundrums include:
* Will I drink coffee or tea?
* Shall I have cake or ice cream?
* Should I use a cache or scratchpad?

While most readers will not face the last choice, it is important for saving time and energy in the devices we love by keeping frequently used information close at hand.

Caches are the workhorse of modern computers, feeding the processor with data about 100X faster than main memory. Decades of hardware research has found clever ways to determine what data to keep in the cache, how to find it, and when to throw it out. The magic of caches is that this cleverness has been hidden almost entirely from software.

But this cleverness costs energy. On every load and store. And it is not always successful.

A scratchpad moves the burden of managing fast accesses to software. Its hardware memory structure is simple and efficient. But its software use is not. Software must explicitly move data in and out of a different scratchpad address space and take responsibility for keeping coherent multiple copies in different address spaces. In practice, this is inefficient, and scratchpads have remained an enticing but niche solution.

Researchers at Illinois and École Polytechnique Fédérale de Lausanne address this conundrum in a paper presented at the International Symposium of Computer Architecture titled “Stash: Have Your Scratchpad and Cache It Too.” Besides offering the proverbial cake as an audience prize, the paper described a new memory organization called stash that gets the best of caches and scratchpads. Stash redistributes the hardware-software burden—software determines what data should go into the stash, but relies on hardware smarts for address space conversion and coherence. Hits in the stash are as efficient as a scratchpad, but (infrequent) misses incur a small hardware penalty. By empowering hardware and software to do what each does best, stash improves both performance and energy.

Stash may not be the final answer, but the paper asks the right question. It also adds to a growing body of recent work imploring hardware and software designers to rethink their distribution of work.