Performance

This toolkit is used to process huge files. As such even a millisecond per operation can add up to minutes and hours. For example, a microsecond over 1 billion operations will add ~16.5 minutes. A millisecond over 1 billion operations will add ~11.5 days.

That's why the performance considerations played the major role in design and implementation of stream-json.

Streams

Every chain in a stream-based data processing pipeline introduces a latency. Try to minimize the size of your pipeline:

• While it is tempting to use a lot of small filters/transforms, try to combine them into one component, if possible (the example use stream-chain for simplicity):

  // fine-grained, but less efficient
  chain([
    sourceStream,
    // filters
    data => data.key % 1 !== 0 ? data : null,
    data => data.value.important ? data : null,
    // transforms
    data => data.value.price,
    price => price * taxRate
  ]);
  
  // more efficient
  chain([
    sourceStream,
    data => {
      if (data.key % 1 !== 0 && data.value.important) {
        return data.value.price * taxRate;
      }
      return null; // ignore
    }
  ]);

  In general, boundaries between streams are relatively expensive, and should be used when stream components generate a varying number of items — this way we can take advantage of stream's ability to handle a back-pressure correctly. Otherwise, simple function calls are more efficient.