multer-s3 upload within express is ridiculously slow

Question

File uploads to S3 via multer-s3 in my express app is ridiculously slow (200kb takes 10 seconds, latency scales in relation to file size). Problem is multer-s3 but can someone tell how to further debug what's happening within the library? Passing onFileUploadStart etc to multer is no longer supported. I have no idea how to peak inside what multer-s3 is doing and in which stages is the latency caused.

Problem is not AWS S3 as uploading from CLI is super-fast, neither is it multer as uploading to local disk is also fast. Any direct S3 javascript operations also work as fast as intended.

Tried turning off multer's content auto type feature but it didn't do much.

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: bucketName,
    acl: 'public-read',
    contentType: multerS3.AUTO_CONTENT_TYPE,
    metadata: function (req, file, cb) {
      cb(null, {fieldName: file.fieldname});
    },
    key: function (req, file, cb) {
      cb(null, Date.now().toString() + "-" + file.originalname);
    }
  },)
})

router.post('/', upload.array('files'), function(req, res, next) {
  console.log('Successfully uploaded ' + req.files.length + ' files!');
});

Want to have sub-second latency as I do with direct AWS CLI upload or using multer to store to local disk, takes 10 seconds to minute or more on sub-MB files.

score 0 · Answer 1 · answered Apr 19 '19 at 22:57

One way to gain insight is to proxy the write function on the response socket, and inspect how its being called:

var upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: bucketName,
    acl: 'public-read',
    contentType: multerS3.AUTO_CONTENT_TYPE,
    metadata: function (req, file, cb) {
      cb(null, {fieldName: file.fieldname});
    },
    key: function (req, file, cb) {
      cb(null, Date.now().toString() + "-" + file.originalname);
    }
  },)
})

router.post('/'
(req, res, next) => { // debug middleware
  const write = res.socket.write;
  res.socket.write = (function(...args) {
    // inspect args, look at the time between invocations of write
    // also check the size of the chunks being passed to write
    return write(...args)
  }).bind(res.socket);
  next();
}
, upload.array('files'), function(req, res, next) {
  console.log('Successfully uploaded ' + req.files.length + ' files!');
});

Thanks, but the 10 second delay is exactly in between the invocation of the upload.array() function and between the first res.socket.write() so this doesn't help. The delay is within the multer-s3 library and I don't know how to debug that. — Villahousut, Apr 20 '19 at 13:45

multer-s3 upload within express is ridiculously slow

1 Answers1