Update docs (#1095)

Author: mthrok

docs/source/best_practice.rst

@@ -4,9 +4,8 @@ Best Practices
 Avoid creating intermediate tensors
 -----------------------------------
 
-For efficient and performant data processing, it is advised to not create
-an intermediate Tensor for each individual media object (such as single image),
-instead create a batch Tensor directly.
+For efficient and performant data processing, avoid creating intermediate Tensors
+for individual media objects (such as single images). Instead, create batch Tensors directly.
 
 We recommend decoding individual frames, then using :py:func:`spdl.io.convert_frames`
 to create a batch Tensor directly without creating an intermediate Tensors.
@@ -33,8 +32,8 @@ separately.
        return spdl.io.to_torch(buffer)
 
 They can be combined in :py:class:`~spdl.pipeline.Pipeline`, which automatically
-discards the items failed to process (for example due to invalid data), and
-keep the batch size consistent by using other items successfully processed.
+discards items that fail to process (for example, due to invalid data) and
+maintains consistent batch size by using successfully processed items.
 
 .. code-block::
 
@@ -76,7 +75,7 @@ might look like the following.
        def __getitem__(self, key: int) -> tuple[Tensor, int]:
            ...
 
-We recommend to separate the source and process and make them additional
+We recommend to separate the source and processing and make them additional
 public interface.
 (Also, as described above, we recommend to not convert each item into
 ``Tensor`` for the performance reasons.)

docs/source/faq.rst

@@ -4,7 +4,7 @@ Frequently Asked Questions
 How to work around GIL?
 -----------------------
 
-In Python, GIL (Global Interpreter Lock) practically prevents the use of multi-threading, however extension modules that are written in low-level languages, such as C, C++ and Rust, can release GIL when executing operations that do not interact with Python interpreter.
+In Python, the GIL (Global Interpreter Lock) practically prevents the use of multi-threading. However, extension modules written in low-level languages such as C, C++, and Rust can release the GIL when executing operations that do not interact with the Python interpreter.
 
 Many libraries used for data loading release the GIL. To name a few;
 
@@ -13,10 +13,9 @@ Many libraries used for data loading release the GIL. To name a few;
 - Decord
 - tiktoken
 
-Typically, the bottleneck of model training in loading and pre-processing the media data.
-So even though there are still parts of pipelines that are constrained by GIL,
-by taking advantage of pre-processing functions that release GIL,
-we can achieve high throughput.
+Typically, the bottleneck in model training is loading and pre-processing media data.
+Even though some parts of pipelines are constrained by the GIL,
+we can achieve high throughput by using pre-processing functions that release the GIL.
 
 What if a function does not release GIL?
 ----------------------------------------
@@ -59,7 +58,7 @@ This will build pipeline like the following.
    ``initializer`` and ``initargs`` arguments.
 
    The values passed as ``initializer`` and ``initargs`` must be picklable.
-   If constructing an object in a process that does not support picke, then
+   If constructing an object in a process that does not support pickle, then
    you can pass constructor arguments instead and store the resulting object
    in global scope. See also https://stackoverflow.com/a/68783184/3670924.
 

docs/source/getting_started/concurrency.rst

@@ -6,7 +6,7 @@ Concurrency
 The pipelines we looked at so far process data sequentially.
 Now let's introduce concurrency to the pipeline so that it finishes jobs faster.
 
-There are two parameters that affects the pipeline performance.
+There are two parameters that affect pipeline performance:
 
 1. Stage concurrency
 2. Thread pool size
@@ -20,9 +20,8 @@ This argument determines at most how many operations of the stage the event loop
 
 .. important::
 
-   Please note that **scheduling multiple tasks concurrently does not necessarily mean
-   all of them are executed concurrently.** The execution of scheduled tasks is subject to
-   the availability of resources required for the execution.
+   Please note that **scheduling multiple tasks concurrently does not guarantee
+   concurrent execution.** Task execution depends on the availability of required resources.
 
    See the :ref:`Thread Pool Size<Thread Pool Size>` for the detail.
 

docs/source/getting_started/intro.rst

@@ -54,7 +54,7 @@ Finally call :py:meth:`Pipeline.stop` to stop the background thread.
 
 It is important to call :py:meth:`Pipeline.stop`.
 Forgetting to do so will leave the background thread running,
-leading to the situation where Python interpreter gets stuck at exit.
+which can cause the Python interpreter to hang at exit.
 
 In practice, there is always a possibility that the application is
 interrupted for unexpected reasons.
@@ -81,8 +81,7 @@ To make sure that the pipeline is stopped, it is recommended to use
 -----------
 
 Unlike processes, threads cannot be killed.
-The ``Pipeline`` object uses a thread pool, and it is important to
-shutdown the thread pool properly.
+The ``Pipeline`` object uses a thread pool, which must be shut down properly.
 
 There are seemingly unharmful patterns, which can cause a deadlock
 at the end of the Python interpreter, preventing Python from exiting.
@@ -178,7 +177,7 @@ at the end of the Python interpreter, preventing Python from exiting.
       # 🚫 Do not keep the iterator object around
       ite = iter(dataloader)
       item = next(ite)
-      # the won't be shutdown won't be shutdown until the `ite` variable
+      # the pipeline won't be shutdown until the `ite` variable
       # goes out of scope. When does that happen??
 
    The ``Pipeline.stop`` is not called until the garbage collector deletes

docs/source/getting_started/parallelism.rst

@@ -5,7 +5,7 @@ Pipeline Parallelism
 
 .. currentmodule:: spdl.pipeline
 
-The :py:class:`Pipeline` supports multi-threading and multi-processing.
+The :py:class:`Pipeline` class supports multi-threading and multi-processing.
 You can also use a ``Pipeline`` objects as source iterator of another ``Pipeline``.
 When experimenting, this flexibility makes it easy to switch multi-threading,
 multi-processing and mixtures of them.
@@ -87,8 +87,7 @@ There are cases where you want to use a dedicated thread for certain task.
    (caching for faster execution or storing the application context)
 #. You want to specify a different number of concurrency.
 
-One notable example that comports with these conditions is transferring a
-data to the GPU.
+One notable example that meets these conditions is transferring data to the GPU.
 Due to the hardware constraints, only one data transfer can be performed
 at a time.
 To transfer data without interrupting the model training,

docs/source/getting_started/stages.rst

@@ -4,7 +4,7 @@ Pipeline Stages
 .. py:currentmodule:: spdl.pipeline
 
 :py:class:`Pipeline` is composed of multiple stages.
-There are mainly three kind of stages.
+There are mainly three kinds of stages.
 
 - Source
 - Processing
@@ -64,12 +64,11 @@ Processing
 Pre-processing is where a variety of operations are applied to the items passed
 from the previous stages.
 
-You can define processing stage by passing an operator function (callable) to
-:py:meth:`~PipelineBuilder.pipe`. (Also there is :py:meth:`~PipelineBuilder.aggregate`
-and :py:meth:`~PipelineBuilder.disaggregate`, which can be used to stack/unstack
-multiple items.)
+You can define a processing stage by passing an operator function (callable) to
+:py:meth:`~PipelineBuilder.pipe`. You can also use :py:meth:`~PipelineBuilder.aggregate`
+and :py:meth:`~PipelineBuilder.disaggregate` to stack and unstack multiple items.
 
-The operator can be either async function or synchronous function.
+The operator can be either an async function or a synchronous function.
 It must take exactly one argument†, which is an output from the earlier
 stage.
 
@@ -136,6 +135,31 @@ and send the decoded frames to GPU asynchronously.
 Sink
 ----
 
-Sink is a buffer where the results of the pipeline is accumulated.
-A sink can be attached to pipeline with :py:meth:`PipelineBuilder.add_sink` method.
+Sink is a buffer where the results of the pipeline are accumulated.
+A sink can be attached to a pipeline with :py:meth:`PipelineBuilder.add_sink` method.
 You can specify how many items can be buffered in the sink.
+
+Advanced: Merging Multiple Pipelines
+-------------------------------------
+
+For more complex data loading scenarios, you can merge outputs from multiple independent
+pipelines using :py:class:`~spdl.pipeline.defs.MergeConfig`. This is useful when you need to:
+
+- Combine data from different sources (e.g., multiple datasets or storage locations)
+- Process different types of data in parallel and merge them downstream
+- Build complex data loading patterns that go beyond linear pipeline structures
+
+The :py:func:`~spdl.pipeline.defs.Merge` function creates a merge configuration that combines
+outputs from multiple :py:class:`~spdl.pipeline.defs.PipelineConfig` objects into a single stream.
+
+.. note::
+
+   The merge mechanism is not supported by :py:class:`PipelineBuilder`. You need to use
+   the lower-level :py:mod:`spdl.pipeline.defs` API and :py:func:`~spdl.pipeline.build_pipeline`
+   function to build pipelines with merge nodes.
+
+.. seealso::
+
+   :ref:`Example: Pipeline definitions <example-pipeline-definitions>`
+      Demonstrates how to build a complex pipeline with merge nodes, including how to
+      combine multiple data sources and process them through a unified pipeline.

docs/source/installation.rst

@@ -25,7 +25,7 @@ The following command will build and install ``spdl`` Python package.
 .. note::
 
    Make sure to use ``-v`` to see the log from the actual build process.
-   The build front end by defaults hide the log of build process.
+   The build front end by default hides the log of build process.
 
 The build process first downloads/builds/installs some third-party
 dependencies, then it builds SPDL and its binding code.

docs/source/overview.rst

@@ -6,7 +6,7 @@ What is SPDL?
 
 SPDL (Scalable and Performant Data Loading) is a library for building
 efficient data preprocessing pipeline, primarily aimed at ML/AI applications.
-It was created by a group of engineers/researchers who works on
+It was created by a group of engineers/researchers who work on
 improving the efficiency of GPU workloads at Meta.
 
 Core Concept
@@ -16,39 +16,39 @@ Its design incorporates the authors' experience on optimizing AI training
 pipelines and the UX/DevX feedbacks from pipeline owners.
 The key features include
 
-- The pipeline construction is intuitive.
-- The pipeline execution is fast and efficient.
-- The pipeline abstraction is flexible so that users can choose structures
-  fit their environment/data/requirements.
+- Pipeline construction is intuitive
+- Pipeline execution is fast and efficient
+- Pipeline abstraction is flexible, allowing users to choose structures that
+  fit their environment, data, and requirements
 - The pipeline can export the runtime statistics of subcomponents, which
   helps identify the bottleneck.
 
-These features allow to create a feedback loop, with which users can
-iteratively improve the performance of the pipeline.
+These features enable a feedback loop that allows users to
+iteratively improve pipeline performance.
 
 .. image:: ./_static/data/spdl_overview_feedback_loop.png
    :width: 480px
 
 Performance & Efficiency
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-Data loading is an important component in AI training. It must be fast
-but also efficient because high CPU utilization can degrade
-the training performance (see :ref:`noisy-neighbour`).
+Data loading is a critical component of AI training. It must be both fast
+and efficient, as high CPU utilization can degrade training performance
+(see :ref:`noisy-neighbour`).
 
 The following plots are from benchmarks we conducted as part of our
 study†.
 
 .. image:: ./_static/data/spdl_overview_performance.png
 
-The figure 5 and 6 show that SPDL if faster than other data loading
+The figure 5 and 6 show that SPDL is faster than other data loading
 solutions, while utilizing CPU more efficiently.
 
 The pipeline abstraction by default uses multi-threading as the core
 parallelism.
 The performance of the pipeline is improved with the recent Python
 version upgrade, and enabling free-threading makes it even faster.
-Aoption of SPDL also paves the way for adoption of free-threaded
+Adoption of SPDL also paves the way for adoption of free-threaded
 Python in ML/AI application.
 
 .. image:: ./_static/data/spdl_overview_speed_vs_version.png

examples/pipeline_definitions.py

@@ -229,7 +229,7 @@ def run_pipeline_example() -> list[int]:
 
     _LG.info("Main pipeline config: %s", main_pipeline_config)
 
-    _LG.info("Builting the pipeline.")
+    _LG.info("Building the pipeline.")
     pipeline = build_pipeline(main_pipeline_config, num_threads=4)
 
     _LG.info("Executing the pipeline.")
@@ -242,7 +242,7 @@ def run_pipeline_example() -> list[int]:
 
 
 def run() -> None:
-    """Run example pipeline and check the resutl."""
+    """Run example pipeline and check the result."""
     results = run_pipeline_example()
 
     _LG.info("Final results: %s", results)