Downloading images from S3 and showing them using SwiftUI results in high memory usage

Question

Hey So I am downloading images from AWS S3 and show them in my app using a swiftUI LazyVGrid.

My code to download is the following:

class S3CacheFetcher: Fetcher {
    
    typealias KeyType = MediaItemCacheInfo
    typealias OutputType = NSData
    
    func get(_ key: KeyType) -> AnyPublisher<OutputType, Error> {
        return download(mediaItem: key).eraseToAnyPublisher()
    }
    
    private func download(mediaItem: KeyType) -> AnyPublisher<OutputType, Error>{
        let BUCKET = "someBucket"
        
        return Deferred {
            Future { promise in
                guard let key:String = S3CacheFetcher.getItemKey(mediaItem: mediaItem) else { fatalError("UserPoolID Error") }
                print("Downloading image with key: \(key)")
                AWSS3TransferUtility.default().downloadData(fromBucket: BUCKET,
                                                            key: key,
                                                            expression: nil) { (task, url, data, error) in
                    if let error = error{
                        print(error)
                        promise(.failure(error))
                    }else if let data = data{
// EDIT--------
                        let encrypt = S3CacheFetcher.encrypt(data: data)
                        let decrypt = S3CacheFetcher.decrypt(data: encrypt)
// EDIT--------
                        promise(.success(decrypt as NSData))

                    }
                }
            }
        }
        .eraseToAnyPublisher()
    }
....
// EDIT---------

// In my code I have a static function that decrypts the images using CryptoKit.AES.GCM

// To test my problem I added these two functions that should stand for my decryption.

    static var symmetricKey = SymmetricKey(size: .bits256)
    static func encrypt(data: Data) -> Data{
        return try! AES.GCM.seal(data, using: S3CacheFetcher.symmetricKey).combined!
    }
    
    static func decrypt(data: Data) -> Data{
        return try! AES.GCM.open(AES.GCM.SealedBox(combined: data), using: S3CacheFetcher.symmetricKey)
    }
}

My GridView:

struct AllPhotos: View {
    @StateObject var mediaManager = MediaManager()
    var body: some View {
      ScrollView{
          LazyVGrid(columns: columns, spacing: 3){
              ForEach(mediaManager.mediaItems) { item in
                  VStack{
                      ImageView(downloader: ImageLoader(mediaItem: item, size: .large, parentAlbum: nil))
                  }
              }
         }
    }
}

My ImageView I am using inside my GridView:

struct ImageView: View{
        @StateObject var downloader: ImageLoader
        
        var body: some View {
            Image(uiImage: downloader.image ?? UIImage(systemName: "photo")!)
                .resizable()
                .aspectRatio(contentMode: .fill)
                .onAppear(perform: {
                    downloader.load()
                })
                .onDisappear {
                    downloader.cancel()
                }
        }
    }

And last but not least the ImageDownloader which is triggered when the image view is shown:

class ImageLoader: ObservableObject {
    @Published var image: UIImage?
    
    private(set) var isLoading = false
    
    private var cancellable: AnyCancellable?
    private(set) var mediaItem:MediaItem
    private(set) var size: ThumbnailSizes
    private(set) var parentAlbum: GetAlbum?
    
    init(mediaItem: MediaItem, size: ThumbnailSizes, parentAlbum: GetAlbum?) {
        self.mediaItem = mediaItem
        self.size = size
        self.parentAlbum = parentAlbum
    }
    
    deinit {
        cancel()
        self.image = nil
    }
    
    func load() {
        guard !isLoading else { return }
        
        // I use the Carlos cache library but for the sake of debugging I just use my Fetcher like below
        cancellable = S3CacheFetcher().get(.init(parentAlbum: self.parentAlbum, size: self.size, cipher: self.mediaItem.cipher, ivNonce: self.mediaItem.ivNonce, mid: self.mediaItem.mid))
            .map{ UIImage(data: $0 as Data)}
            .replaceError(with: nil)
            .handleEvents(receiveSubscription: { [weak self] _ in self?.onStart() },
                          receiveCompletion: { [weak self] _ in self?.onFinish() },
                          receiveCancel: { [weak self] in self?.onFinish() })
            .receive(on: DispatchQueue.main)
            .sink { [weak self] in self?.image = $0 }
    }
    
    func cancel() {
        cancellable?.cancel()
        self.image = nil
    }
    
    private func onStart() {
        isLoading = true
    }
    
    private func onFinish() {
        isLoading = false
    }
}

So first of all before I describe my problem. Yes I know I have to cache those images for a smother experience. I did that but for the sake of debugging my memory issue I do not cache those images for now.

Expected behavior: Downloads images and displays them if the view is shown. Purges the images out of memory if the image view is not shown.

Actual behavior: Downloads the images and displays them but it does not purge them from memory once the image view has disappeared. If I scroll up and down for some period of time the memory usage is up in the Gb range and the app crashes. If I use my persistent cache which grabs the images from disk with more or less the same logic for grabbing and displaying the images than everything works as expected and the memory usage is not higher than 50 Mb.

I am fairly new to Combine as well as SwiftUI so any help is much appreciated.