Open
Conversation
VaggelisD
changed the title
VaggelisD
force-pushed
the
mypyc-separate-flag
branch
from
071ea34 to
8db1da4
Compare
…[CLAUDE] sqlglot-mypy 1.20.0.post3 makes mypyc's separate=True compilation work correctly on real-world projects (cross-group class inheritance, generator helper classes, non-ext subclasses with fast methods, mutually-dependent compiled modules). Turning it on for sqlglotc cuts incremental rebuild time dramatically — a 1-file edit goes from ~110s (full monolithic rebuild) to ~3s. Three small changes: - sqlglotc/setup.py: pass separate=True to mypycify(). Produces one shared lib + shim pair per module instead of one monolithic shared lib; mypyc only regenerates the changed module's C on rebuild. - sqlglot/__init__.py: tighten the `*__mypyc.so` bootstrap preloader. Under separate=True, per-module shared libs (e.g. errors__mypyc.so) live next to their .py sources and are resolved via normal dotted imports through the shim. Skip preloading those; keep the legacy preload behavior only for monolithic-hash .so files that lack a .py sibling. - sqlglot/optimizer/__init__.py: make the top-level re-exports lazy via PEP 562 __getattr__. Under separate=True's eager cross-group init, the previous eager `from sqlglot.optimizer.optimizer import ...` could trigger while sqlglot's own __init__.py was mid-flight, hitting a circular-import ImportError on `from sqlglot import Schema, exp`. Lazy imports defer the cycle to first use.
VaggelisD
force-pushed
the
mypyc-separate-flag
branch
from
8db1da4 to
e8129f7
Compare
tobymao
reviewed
Apr 24, 2026
|
|
||
| from __future__ import annotations | ||
|
|
||
| import typing as _t |
Owner
There was a problem hiding this comment.
this shouldn't be _t, it's t right
tobymao
reviewed
Apr 24, 2026
| import typing as _t | ||
|
|
||
| if _t.TYPE_CHECKING: | ||
| from sqlglot.optimizer.optimizer import RULES as RULES, optimize as optimize |
tobymao
reviewed
Apr 24, 2026
| trigger this package's ``__init__.py`` before ``sqlglot`` has finished | ||
| binding names like ``exp`` / ``Schema``. Eager ``from sqlglot.optimizer.optimizer | ||
| import ...`` at module top used to kick off a circular-import cascade. | ||
| PEP 562 ``__getattr__`` lets the names resolve only when they're first |
Owner
There was a problem hiding this comment.
can you dig into this a little bit more, is it actually true? is the fix actually in mypyc?
tobymao
reviewed
Apr 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Switch
sqlglotc's mypyc build toseparate=True, plus the two small sqlglot-side tweaks needed to make it work end-to-end.The
separate=Trueflag gives each compiled module its own shared lib + shim, so mypyc only has to regenerate / recompile the modules whose source actually changed. Clean builds are roughly the same wall-clock; incremental rebuilds after a one-line edit drop dramatically once the cache is warm.Three small changes, all behind the
[c]compile path:sqlglotc/setup.py: passseparate=Truetomypycify(). Everything else in the build recipe is unchanged.sqlglot/__init__.py: the existing*__mypyc*.sobootstrap preloader was written for the old monolithic build where a single hash-named.sosat at the package root. Underseparate=Truethe per-module shared libs (e.g.errors__mypyc.so) live next to their.pysiblings and resolve through Python's normal import machinery via the shim. Skip preloading those; keep the legacy behavior only for.sofiles that don't have a.pysibling.sqlglot/optimizer/__init__.py: swap the eager top-level re-exports for PEP 562 lazy__getattr__. Underseparate=True's cross-group init ordering, the previous eagerfrom sqlglot.optimizer.optimizer import ...could fire whilesqlglot's own__init__.pywas still mid-bootstrap, hitting a circular-importImportErroronfrom sqlglot import Schema, exp. Lazy imports defer the cycle to first use.