0

Task: Upload a large(!) user text input from a html/vue client to a node.js server in a performant way.

Attempted (currently using) solution:

Im using a form input (textarea) and just send it like this (simplified):

  <form action="/loadWrittenText" method="post">
      <textarea type="text" rows="15" id="textbox" name="textInput">
      </textarea>
       <button>Analyze</button>
  </form>

and just process it like a normal post request: 

  router.post('/loadWrittenText', function (req, res) {
      wait.launchFiber(postLoadWrittenText, req, res, req);
  });

Problem:

If the user is attempting to input large texts it would first of all run int this problem: Error: request entity too large

Secondly I want to attempt upload very big texts (atleast 10.000 words - entire books if possible). At some point I think a simple post request cant do the job (atleast that is my guess).

Also I experience a high use of RAM if I try that, but this can be caused something else too.

Other Solutions I thought of:

-Generally using some kind of stream where I input the text and it gets streamed to the server and processed chunk for chunk. I couldnt find a good way to do this with text inputs though.

-Using a file uploader like this one: https://coligo.io/building-ajax-file-uploader-with-node/ (Problem with that would be that I still would have a user text input and a file input so what happens if the user decides to use the text input and types a very big text? I dont want to throw away the text-input if possible)

-kick the router.post part and replace the whole thing with socket.io (I'm no pro with socket.io though, so would need a little bit of code to get the idea)

Context:

Im building an app that semantically analyses texts. Basically the user will input text, it gets analysed, seved on a Database and send back to the user. The whole thing works but I need to scale it for bigger texts and also making it more performant.

Resources I use that might be important for the solution:

-node.js express-server -express-vue -> vue (frontend) -wait.for (that why that wait.launchFiber line)

telion
  • 582
  • 3
  • 22

2 Answers2

0

Instead of sending a <textarea> value, try using a <input type="file"> and stream the file content directly to your server.

https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/file

Here's a good resource if you need help with node streams:

https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93

rafaelcastrocouto
  • 10,518
  • 1
  • 34
  • 58
  • I found out about the type="file" too, but is there no way to get the user typed input on the server even if its a lot? But thank you for that answer. – telion Feb 27 '18 at 14:31
  • I guess the best approach would be to slice the data somehow. Since you can have a user in a very low memory device, smaller data would make all UI more responsive and would also avoid data loss. – rafaelcastrocouto Feb 27 '18 at 15:17
0

Try following code snippet.

app.use(express.bodyParser({limit: '50mb'}));
Sagar
  • 967
  • 7
  • 15
  • If I understood it correctly I get around the browser exception with that. But if I load a book with maybe 1gb, I guess raising the limit on a request wont do the trick, does it? – telion Feb 27 '18 at 14:26