If your use case is lots of images/icons in a JList/JTable inside a JScrollPane, I once wrote something similar for a CMS search result display:
I had thumbnails of larger images stored on server/cached on disk. You can exploit the fact that the JList/JTable (or Swing components in general) will only try to paint the components visible on screen, so you don't need to track this yourself. I used a JLabel with ImageIcon as renderer in the list, and a fixed size cache of icons in memory. Whenever the renderer hit an image not in cache, I used a lazy-loading technique with a placeholder icon returned immediately, while loading the icon in the background using a SwingWorker. When loaded, the SwingWorker issued a repaint to JList. The icon would now be in cache, and the list would paint nicely.
Some bonus tips: Make sure your icon cache is larger than the maximum number of items that can be displayed on screen. Also, use a limited (fixed size) thread pool for your SwingWorkers (possibly loading multiple icons at once), to avoid exessive reapint-loops that may cause flickering.