0

This one is a bit dense. I'm building a web-socket based FUSE filesystem in Node.js (v14.14.0) using the fuse-native package.

To transfer the file data between the client and the server, I'm using the websocket-stream package to stream the binary data back and forth.

This works fine when transferring a file from the server to the client, but when trying to transfer a file from the client to the server, I'm running into a problem.

fuse-native passes around Buffer instances with binary segments of the file being transferred. I'm trying to write the Buffer to a websocket-stream stream and receive it on the server, where it will be streamed to a temporary file.

Here's how this happens. On the client-side, the following method is called:

write(buffer) {
    console.log('write buffer pre slice', buffer)

    const write_stream = WebSocketStream(`ws://localhost:5746/?socket_uuid=${this.socket_uuid}&node_uuid=${this.node_uuid}&length=${this.length}&position=${this.position}&writing_file=true`, {
        perMessageDeflate: false,
        binary: true,
    })

    console.log(write_stream)
    console.log('writing buffer', buffer.toString(), buffer)

    write_stream.push(buffer)
    write_stream.push(null)
}

According to the Node.js docs, I should be able to pass the Buffer directly to the stream. However, on the server, no data is ever received. Here's how the server is receiving:

async on_file_write_stream(stream, request) {
    let { socket_uuid, node_uuid, length = 4096, position = 0 } = url.parse(request.url, true).query
    if ( typeof position === 'string' ) position = parseInt(position)
    if ( typeof length === 'string' ) length = parseInt(length)
    const socket = this.sockets.find(x => x.uuid === socket_uuid)

    if ( !socket.session.temp_write_files ) socket.session.temp_write_files = {}

    const placeholder = socket.session.temp_write_files?.[node.uuid] || await tmp.file()
    socket.session.temp_write_files[node.uuid] = placeholder

    console.log('Upload placeholder:', placeholder)
    console.log('write stream', stream)
    console.log('write data', { placeholder, position, length })

    stream.pipe(fs.createWriteStream(placeholder.path, { flags: 'r+', start: position }))
}

Once the client-side code finishes, the temporary file is still completely empty. No data is ever written.

The strange part is that when I cast the buffer to a string before writing it to the stream (on the client side), all works as expected:

write(buffer) {
    console.log('write buffer pre slice', buffer)

    const write_stream = WebSocketStream(`ws://localhost:5746/?socket_uuid=${this.socket_uuid}&node_uuid=${this.node_uuid}&length=${this.length}&position=${this.position}&writing_file=true`, {
        perMessageDeflate: false,
        binary: true,
    })

    console.log(write_stream)
    console.log('writing buffer', buffer.toString(), buffer)

    write_stream.push(buffer.toString())
    write_stream.push(null)
}

This works fine for text-based files, but binary files become mangled when transferred this way. I suspect it's because of the string cast before transfer.

How can I send the buffer data along the stream without casting it to a string first? I'm not sure why the server-side stream isn't receiving data when I write the Buffer directly.

Thanks in advance.

For the curious, here is the full server-side file and the client-side file.

glmdev
  • 341
  • 4
  • 8
  • U would be able to pipe data dire you to the websocket stream. Take a look at this example https://stackoverflow.com/questions/22723671/streaming-data-in-node-js-with-ws-and-websocket-stream – Bojoer Nov 27 '20 at 03:10
  • Because of the FUSE implementation, I have to read the data from a Buffer on the client-side. If the websocket-stream stream actually receives data on the server, it writes to the file just fine. – glmdev Nov 27 '20 at 03:19
  • You should then be able to convert the buffer to a stream using combined stream? Take a look at this example it should do the trick I think https://exceptionshub.com/nodejs-create-stream-from-buffer.html – Bojoer Nov 27 '20 at 04:05
  • Hm. Just tried it, but it seems to have the same issue as when I pass the buffer directly. No data is written on the server-side. – glmdev Nov 27 '20 at 04:32

1 Answers1

1

Kind of a hack, but I worked around the problem by base64 encoding the Buffer on the client-side and decoding it on the server side.

Thanks to user Bojoer for pointing me in the direction of the combined-stream library, which I used to pipe the Buffer to the stream:

const combined_stream = CombinedStream.create()
combined_stream.append(buffer.toString('base64'))
combined_stream.pipe(write_stream)

Then, decode it on the server-side:

const encoded_buffer = await this._bufferStream(stream)
const decoded_buffer = new Buffer(encoded_buffer.toString(), 'base64')

console.log({encoded_buffer, decoded_buffer})

const combined_stream = CombinedStream.create()
combined_stream.append(decoded_buffer)
combined_stream.pipe(fs.createWriteStream(placeholder.path, { flags: 'r+', start: position }))

This is sub-optimal, though, as converting back and forth from base64 takes processing, and it imposes a bandwidth penalty. I'm still curious about the original problem why I can't write a binary Buffer to the websocket stream. Maybe it's a limitation of the library.

glmdev
  • 341
  • 4
  • 8