Utilizing pickled environments to optimize build time

[jptelle]

Hello all,

First off, thank you for developing and maintaining such an incredible utility. My team uses this extension to its fullest capacity and we have had great results so far. Please scroll to the bottom of the post if TLDR to see the emboldened question.

Our software is composed entirely of Matlab code that is heavily abstracted and distributed in several packages. There are nearly 2k scripts, and the size of the project has warranted the development of a proprietary utility for parsing out the codebase and generating the rST file structure before building with Sphinx. Everything is documented using the automodule, autoclass, autofunction, etc. directives.

In the day-to-day development process, our team spends a considerable amount of time formatting comments within the Matlab scripts such that they render properly in the generated documentation. We've noticed that, upon calling sphinx-build, the sphinxcontrib-matlabdomain extension files through and processes every script that is referenced in the index. This is indicated by the following output in the build log:

Running Sphinx v8.2.3
[sphinxcontrib-matlabdomain] matlab_short_links=True, forcing matlab_keep_package_prefix=False.
loading pickled environment: The configuration has changed (22 options: 'applehelp_bundle_name', 'applehelp_kb_product', 'applehelp_title', 'author', 'datatables_version', ...)
done
building [mo]... targets for 0 po files that are out of date
writing output
building [html]... targets for 345 source files that are out of date
updating environment... [config changed ('project')] 345 added, 0 changed, 0 removed

There are a total of 345 rST files that contain directives referencing all of the scripts in our codebase. Thus, the output indicates that all of the source files are "out of date". Even if we build the documentation twice consecutively with zero changes to the source files, the same behavior ensues. Functionally, there is no problem with this. However, it does add considerable overhead to the build time and thus significantly increases the time required to document scripts, since the developer often checks their work throughout the process. The compilation time varies by machine, but it has been reported in the range of 1 to 10 minutes.

We are looking to implement a "preview" feature for the purpose of trimming down this build time, but the best solution we've proposed thus far seems to be a bit hacky, e.g.:

1. Create a temporary "micro-project" in the OS-specific temp directory
2. Copy just the Matlab file in question to the source directory
3. Build the micro-project with Sphinx
4. Transfer the corresponding generated file to the parent docs directory and view

This method has an obvious downside in that interlinking will not work for the previewed document. It is merely intended as a stopgap solution to ensure formatting is correct before building the entire codebase documentation.

TLDR: My question is: Is enabling compatibility of this extension with Sphinx pickled environments feasible, such that it can detect when referenced scripts have changes and build accordingly? If not, would a workaround be worth considering, such as a hook for specific paths to scripts that need to be processed/omitted?

Hopefully I've contextualized the problem sufficiently. I understand this is a bit beyond the pale in terms of the use case of this extension, but your consideration is much appreciated. Thank you for your time, and please let me know if you require more information.

[jasonnicholson]

You should use sphinx-autobuild. You don't have to create any preview. You fire up sphinx-autobuild pointed at buidling the one rST file you want and then the rebuild is fast. Our projects are the on the order as the same size as your project. We build a folder of 10+-ish files in 5ish seconds. It works well.

https://github.com/sphinx-doc/sphinx-autobuild

[jmgate]

@jasonnicholson, could you please provide more details on how you use sphinx-autobuild with your MATLAB repository? Our layout is as follows:

repository-root/
└── Source/
    ├── Documents/
    │   └── sphinx/
    │       └── api/  # contains RST files corresponding to the *.m files in Source/Matlab
    └── Matlab/  # contains all our MATLAB packages/scripts/etc.

Within the sphinx directory, I can run

sphinx-autobuild -b html --watch ../../Matlab/path/to/package . _build api/path/to/package/package.rst

and that is convenient, because any time I change a *.m file in Matlab/path/to/package, Sphinx automatically rebuilds, only writes the affected HTML files, and the HTML is automatically reloaded in my browser. However, Sphinx + matlabdomain still reads and processes all of our code under Source/Matlab, and that's the significant time sink. Is there something I'm missing?

[jasonnicholson]

So, I ended up hard linking only the m files I am processing. This means only a few of the m files are processed.

[jasonnicholson]

I have time to explain. Here is the longer version.

We put markdown files next to our source code so we custom build the docsource (Sphinx source folder) from a template docsource and the code folder.

repository-root/
└── Source/
    ├── Documents/
    │   └── docsource/
    │       └── api/  # contains RST files corresponding to the *.m files in Source/Matlab
    │       └── code/ # hard link and folders pointing to Source/MATLAB. This includes m-files and .md files.
    │       └── conf.py
    └── Matlab/  # contains all our MATLAB packages/scripts/etc.

We have a function that builds the docsource folder above with the following inputs:

• Array of string patterns corresponding to source code, (:,1) string. An empty array builds everything, "". A single string(missing) builds only the template files.
• Default exclusion patterns. This is a list of 28 paths that don't currently work with the matlab-domain plugin for various reasons.

If we built all code files, we would have 2000+ files. We have our own auto generated rst files based on what we got from the previous inputs. Code files are hard linked (while maintain folder structure) to the code folder so that when modified in their original location, they update in the other location. With sphinx-autobuild, this means an extra --watch pointing to where the code resides on disk.

In the conf.py, we use a relative path that points at the code folder.

Overall, what we have built got us to nice looking documentation and 5-10 second build times with sphinx-autobuild. We usually focus on a folder at a time when authoring which gives 5 second build times of 5-10 m-files.

Our calls the auto build look like below. We are on Windows.

start "Sphinx Auto Build Server" "sphinx-autobuild" --port 9326 --open-browser --watch "D:\original\source\location" -b html --quiet "D:\path\to\docsite\docsource" "D:\path\to\docsite\build\html"