Upload to S3 using multer without a middleware function

Question

I am using multer to upload media to my s3 bucket. I am using multer-s3 as a middleware to upload media like:

const upload = multer({
  storage: multerS3({
    s3: s3,
    bucket: myBucket,
    key: function (req, file, cb) {
      cb(null, new Date().getTime() + '_' + file.originalname)
    }
  })
});

and calling it in route as:

router.post("/media",upload.single("media"))

This works great. But a scenario that is not working for me is: Let say i upload one image and i want to store multiple version of it by resizing it before uploading. I am not able to call the upload function like a normal one. I want to do something like:

let thumbnail = myFunctionToReturnImageFile(req.file);
upload(thumbnail);

I understand i need to send some multi-part/form-part but i am unable to find a solution. If you suggest me a something that will be great.

I am trying to do something similar and I think you could use MemoryStorage, modify the image and then upload. — Javier Guzmán, Nov 21 '20 at 06:19
@JavierGuzmán I will apprietiate if you can share some code samples — Muhammad Zeeshan, Nov 21 '20 at 06:21
I cannot share any code examples because I am currently trying something similar and I have do not have anything working yet. However, I am using this link as a reference: https://medium.com/@olamilekan001/image-upload-with-google-cloud-storage-and-node-js-a1cf9baa1876 — Javier Guzmán, Nov 21 '20 at 08:23

Simone Nigro · Answer 1 · 2020-12-10T20:59:16.380

3

Multer is a busboy wrapper. You can use busboy instead of multer for handle the request and make all operations you need.

const Busboy = require('busboy');

router.post('/media', function(req, res) {

    var busboy = new Busboy({ headers: req.headers });

    busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {

      console.log('File ' + filename + ' upload incoming');

      
      file.on('end', function() {
         console.log('File ' + filename + ' upload finished');

         // make all you want, eg:
         let thumbnail = myFunctionToReturnImageFile(file);
         upload(thumbnail);

         res.send('file uploaded');
      });
    });
});

edited Dec 10 '20 at 20:59

answered Nov 22 '20 at 09:16

Simone Nigro

3,928
2
24
55

I know this answers "without a middleware function" to the letter, but I don't think there's any reasonable need not tu use multer (or simillar mw) to parse `req.files` and make use of them in later function... – ΔO 'delta zero' Nov 28 '20 at 20:27
hi, i have a very similar question can you please have a look at. https://stackoverflow.com/questions/65479245/nodejs-multer-aws-s3 – kd12345 Dec 29 '20 at 09:55

score 3 · Accepted Answer · answered Nov 23 '20 at 15:15

Option 1 - Use multer-s3-transform middleware to handle S3 upload with file transformations, like image resizing using sharp.

const multer = require('multer')
const multerS3 = require('multer-s3-transform')
const sharp = require('sharp')
const AWS = require('aws-sdk')
const S3 = new AWS.S3({
    accessKeyId: ...,
    secretAccessKey: ...
})

const upload = multer({
  storage: multerS3({
    s3: S3,
    bucket: ...,
    shouldTransform: true,
    transforms: [
      {
        id: 'original',
        key: (req, file, cb) => cb(null, new Date().getTime() + '_' + req.file.originalname),
        transform: (req, file, cb) => cb(null, sharp().jpg())
      },
      {
        id: 'large',
        key: (req, file, cb) => cb(null, new Date().getTime() + '_large_' + req.file.originalname),
        transform: (req, file, cb) => cb(null, sharp().resize(1200, 900).jpg())
      },
      {
        id: 'small',
        key: (req, file, cb) => cb(null, new Date().getTime() + '_small_' + req.file.originalname),
        transform: (req, file, cb) => cb(null, sharp().resize(400, 300).jpg())
      }
    ]
  })
})

router.post('/media', upload.single('media'))

Option 2 - If you insist on doing things manually, you may use plain multer to handle the incoming file, resize it with sharp and upload each resized file to S3 using AWS-SDK.

const multer = require('multer')
const sharp = require('sharp')
const AWS = require('aws-sdk')
const S3 = new AWS.S3({
    accessKeyId: ...,
    secretAccessKey: ...
})

router.post('/media', multer().single('media'), (req, res) => {
  // req.file represents the uploaded file
  let time = new Date().getTime()
  
  Promise.all([
    // original file
    S3.upload({
      Bucket: ...,
      Key: time + '_' + req.file.originalname,
      Body: req.file.buffer
    }),
    
    // large preview
    sharp(req.file)
      .resize(1200, 900)
      .toBuffer()
      .then(resized => S3.upload({
        Bucket: ...,
        Key: time + '_large_' + req.file.originalname,
        Body: resized
      }).promise()),
    
    // small thumbnail
    sharp(req.file)
      .resize(400, 300)
      .toBuffer()
      .then(resized => S3.upload({
        Bucket: ...,
        Key: time + '_small_' + req.file.originalname,
        Body: resized
      }).promise()),
  ])
  .then(() => res.send("Images uploaded"))
  .catch(e => {
    console.warn(e) // debug this error
    res.status(500).send("Unable to upload images")
  })
})

PS: this is a blind code, hope there are not too many typos. — ΔO 'delta zero', Nov 23 '20 at 16:03
Hi, which do you suggest option 1 or option 2. I tried option 2, but when i checked the images on aws they were not resized. So i went with option 2 and it worked but i dont understand why. What is the main difference between both? — kd12345, Dec 28 '20 at 14:04
@kd12345. I'd recommend option 1. Opt 2 gives you more flexibility, but unless you need it, opt 1 gives cleaner code. — ΔO 'delta zero', Dec 29 '20 at 11:59
Hi, can you please have a look at my question here, would really appreciate the help. https://stackoverflow.com/questions/65479245/nodejs-multer-aws-s3 — kd12345, Dec 29 '20 at 12:04
@kd12345, that looks a bit messy. Try to implement this code (opt 1) as it is and build from there. — ΔO 'delta zero', Dec 29 '20 at 13:47
Hi, did exactly as you suggested making the code look better and implementing option 1, but now the files do not have the same Date each one has a different date — kd12345, Dec 29 '20 at 13:55

score 0 · Answer 3 · answered Nov 21 '20 at 07:55

0

You can try using Node.js image processing like sharp to resize then upload to S3.

sharp(req.file)
  .resize(320, 240)
  .toBuffer()
  .then(thumbnail => { upload(thumbnail) })
  .catch( err => { ... });

answered Nov 21 '20 at 07:55

Méo

86
4

The problem here is i can't access upload method like you used. It is only offered to be used as a middleware. – Muhammad Zeeshan Nov 21 '20 at 07:57
@muhammad-zeeshan I think you should write a middleware wrap resize and upload function. – Méo Nov 21 '20 at 08:05

Upload to S3 using multer without a middleware function

3 Answers3