72 lines
2.0 KiB
Markdown
72 lines
2.0 KiB
Markdown
|
through2-concurrent
|
||
|
|
===================
|
||
|
|
|
||
|
|
[](https://nodei.co/npm/through2-concurrent/)
|
||
|
|
|
||
|
|
A simple way to create a Node.JS Transform stream which processes in
|
||
|
|
parallel. You can limit the concurrency (default is 16) and order is
|
||
|
|
*not* preserved (so chunks/objects can end up in a different order to
|
||
|
|
the order they started in if the transform functions take different
|
||
|
|
amounts of time).
|
||
|
|
|
||
|
|
Built using [through2](https://github.com/rvagg/through2) and has the
|
||
|
|
same API with the addition of a `maxConcurrency` option.
|
||
|
|
|
||
|
|
Non-`objectMode` streams are supported for completeness but I'm not
|
||
|
|
sure they'd be useful for anything.
|
||
|
|
|
||
|
|
Written by Thomas Parslow
|
||
|
|
([almostobsolete.net](http://almostobsolete.net) and
|
||
|
|
[tomparslow.co.uk](http://tomparslow.co.uk)) as part of Active Inbox
|
||
|
|
([activeinboxhq.com](http://activeinboxhq.com/)).
|
||
|
|
|
||
|
|
[](https://travis-ci.org/almost/through2-concurrent)
|
||
|
|
|
||
|
|
|
||
|
|
Install
|
||
|
|
-------
|
||
|
|
|
||
|
|
```bash
|
||
|
|
npm install --save through2-concurrent
|
||
|
|
```
|
||
|
|
|
||
|
|
Examples
|
||
|
|
--------
|
||
|
|
|
||
|
|
Process lines from a CSV in paralel. The order the results end up in
|
||
|
|
the `all` variable is not deterministic.
|
||
|
|
|
||
|
|
```javascript
|
||
|
|
var through2Concurrent = require('through2-concurrent');
|
||
|
|
|
||
|
|
var all = [];
|
||
|
|
|
||
|
|
fs.createReadStream('data.csv')
|
||
|
|
.pipe(csv2())
|
||
|
|
.pipe(through2Concurrent.obj(
|
||
|
|
{maxConcurrency: 10},
|
||
|
|
function (chunk, enc, callback) {
|
||
|
|
var self = this;
|
||
|
|
someThingAsync(chunk, function (newChunk) {
|
||
|
|
self.push(newChunk);
|
||
|
|
callback();
|
||
|
|
});
|
||
|
|
}))
|
||
|
|
.on('data', function (data) {
|
||
|
|
all.push(data)
|
||
|
|
})
|
||
|
|
.on('end', function () {
|
||
|
|
doSomethingSpecial(all)
|
||
|
|
})
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
Contributing
|
||
|
|
------------
|
||
|
|
|
||
|
|
Fixed or improved stuff? Great! Send me a pull request [through GitHub](http://github.com/almost/through2-concurrent)
|
||
|
|
or get in touch on Twitter [@almostobsolete][#tom-twitter] or email at tom@almostobsolete.net
|
||
|
|
|
||
|
|
[#tom]: http://www.almostobsolete.net
|
||
|
|
[#tom-twitter]: https://twitter.com/almostobsolete
|