This one is a bit dense. I'm building a web-socket based FUSE filesystem in Node.js (v14.14.0) using the fuse-native package.
To transfer the file data between the client and the server, I'm using the websocket-stream package to stream the binary data back and forth.
This works fine when transferring a file from the server to the client, but when trying to transfer a file from the client to the server, I'm running into a problem.
fuse-native passes around Buffer instances with binary segments of the file being transferred. I'm trying to write the Buffer to a websocket-stream stream and receive it on the server, where it will be streamed to a temporary file.
Here's how this happens. On the client-side, the following method is called:
write(buffer) {
console.log('write buffer pre slice', buffer)
const write_stream = WebSocketStream(`ws://localhost:5746/?socket_uuid=${this.socket_uuid}&node_uuid=${this.node_uuid}&length=${this.length}&position=${this.position}&writing_file=true`, {
perMessageDeflate: false,
binary: true,
})
console.log(write_stream)
console.log('writing buffer', buffer.toString(), buffer)
write_stream.push(buffer)
write_stream.push(null)
}
According to the Node.js docs, I should be able to pass the Buffer directly to the stream. However, on the server, no data is ever received. Here's how the server is receiving:
async on_file_write_stream(stream, request) {
let { socket_uuid, node_uuid, length = 4096, position = 0 } = url.parse(request.url, true).query
if ( typeof position === 'string' ) position = parseInt(position)
if ( typeof length === 'string' ) length = parseInt(length)
const socket = this.sockets.find(x => x.uuid === socket_uuid)
if ( !socket.session.temp_write_files ) socket.session.temp_write_files = {}
const placeholder = socket.session.temp_write_files?.[node.uuid] || await tmp.file()
socket.session.temp_write_files[node.uuid] = placeholder
console.log('Upload placeholder:', placeholder)
console.log('write stream', stream)
console.log('write data', { placeholder, position, length })
stream.pipe(fs.createWriteStream(placeholder.path, { flags: 'r+', start: position }))
}
Once the client-side code finishes, the temporary file is still completely empty. No data is ever written.
The strange part is that when I cast the buffer to a string before writing it to the stream (on the client side), all works as expected:
write(buffer) {
console.log('write buffer pre slice', buffer)
const write_stream = WebSocketStream(`ws://localhost:5746/?socket_uuid=${this.socket_uuid}&node_uuid=${this.node_uuid}&length=${this.length}&position=${this.position}&writing_file=true`, {
perMessageDeflate: false,
binary: true,
})
console.log(write_stream)
console.log('writing buffer', buffer.toString(), buffer)
write_stream.push(buffer.toString())
write_stream.push(null)
}
This works fine for text-based files, but binary files become mangled when transferred this way. I suspect it's because of the string cast before transfer.
How can I send the buffer data along the stream without casting it to a string first? I'm not sure why the server-side stream isn't receiving data when I write the Buffer directly.
Thanks in advance.
For the curious, here is the full server-side file and the client-side file.