Home » Nodejs » What is a correct way to pause piped readable stream from writeable one in nodejs?

What is a correct way to pause piped readable stream from writeable one in nodejs?

Posted by: admin November 30, 2017 Leave a comment

Questions:

I am writing a module, which is a writeable stream. I want to implement pipe interface for my users.

If some error happens, i need to pause readable stream and emit error event. Then, user will decide – if he is ok with error, he should be able to resume to data processing.

var writeable = new BackPressureStream();
writeable.on('error', function(error){
    console.log(error);
    writeable.resume();
});

var readable = require('fs').createReadStream('somefile.txt');
readable.pipe.(writeable);

I see that node provides us with readable.pause() method, that can be used to pause readable stream. But i can’t get how i can call it from my writeable stream module:

var Writable = require('stream').Writable;

function BackPressureStream(options) {
    Writable.call(this, options);
}
require('util').inherits(BackPressureStream, Writable);

BackPressureStream.prototype._write = function(chunk, encoding, done) {
    done();
};

BackPressureStream.prototype.resume = function() {
    this.emit('drain');
}

How back pressure can be implemented in a writeable stream?

P.S. It is possible to use pipe/unpipe events, that provide readable stream as a parameter. But it is also said, that for piped streams, the only chance to pause is to unpipe readable stream from writeable.

Did i got it right? I have to unpipe my writeable stream until user calls resume? And after user calls resume, i should pipe readable stream back?

Answers:

Basically, as I understand it, you are looking to put backpressure on the stream in the case of an error event. You have a couple of options.

Firstly, as you have already identified, use pipe to grab an instance of the read stream and do some fancy footwork.

Another option is to create a wrapping writable stream that provides this functionality (i.e. it takes a WritableStream as an input, and when implementing stream functions, passes the data along to the supplied stream.

Basically you end up with something like

source stream -> wrapping writable -> writable

https://nodejs.org/api/stream.html#stream_implementing_a_writable_stream deals with implementing a writable stream.

The key for you is that if an error occurs in the underlying writable, you would set a flag on the stream, and the next call to write to happen, you would buffer the chunk, store the callback and only call. Something like

// ...
constructor(wrappedWritableStream) {
    wrappedWritableStream.on('error', this.errorHandler);
    this.wrappedWritableStream = wrappedWritableStream;
}
// ...
write(chunk, encoding, callback) {
    if (this.hadError) {
        // Note: until callback is called, this function won't be called again, so we will have maximum one stored
        //  chunk.
        this.bufferedChunk = [chunk, encoding, callback];
    } else {
        wrappedWritableStream.write(chunk, encoding, callback);
    }
}
// ...
errorHandler(err) {
    console.error(err);
    this.hadError = err;
    this.emit(err);
}
// ...
recoverFromError() {
    if (this.bufferedChunk) {
        wrappedWritableStream.write(...this.bufferedChunk);
        this.bufferedChunk = undefined;
    }
    this.hadError = false;
}

Note: You should only need to implement the write function, but I do encourage you to dig around and play with the other implementation functions.

It is also worth noting that you may have some trouble writing to streams which have emitted an error event, but I will leave that to you as a separate problem to solve.

Here’s another good resource on backpressuring
https://nodejs.org/en/docs/guides/backpressuring-in-streams/