render_elixir_function_defs/3 reads a module's whole IR out of the PLT, prunes it to the functions the page reaches, and encodes them. That happens once per page, per module, and encode_ir/2 for a module definition is already a per-function fold, so the same function is encoded again for every page that reaches it.
Measured on an app with 9,018 modules and 158 pages: 207,292 function encodings, 3,810 of them distinct. Every function is encoded about 54 times on average.
The reads cost more than the encodes. ETS copies on read, and pages reach about 4.5 functions per module, so a page pulls a whole module IR, up to 55MB, to keep a handful of functions and discard the rest. Over one compile that moves 213GB out of the PLT, and create_page_entry_files spends 131 seconds of a 201 second compile there, which works out to roughly memory bandwidth.
Worth knowing for protocol modules: maybe_prune_protocol_dispatcher_function_defs/3 selects their function defs against the whole page's reachable set rather than the module's own MFAs, so which functions a page includes is page-dependent there even though the encoding of each one is not.
Measurements
Same app, prototype keyed on {module, kept mfas} rather than per function:
| phase |
before |
after |
create_page_entry_files |
130.9s |
48.3s |
| total compile |
201s |
130s |
Cache held 1,143 entries, 59,162 hits against 1,527 misses, about 45MB as refc binaries. The misses exceed the entry count because concurrent tasks miss the same key before either inserts.
That coarser key measured well here only because 830 of the 844 reachable modules produce one variant across all pages. It degrades on apps whose pages reach different fragments of a module, where a {module, function, arity} key does not.
For comparison, moving the IR PLT to :persistent_term for zero-copy reads gets create_page_entry_files to 50.5s, but needs +MIscs raised at VM start (the default 1GB literal carrier cannot hold a 2.6GB IR table), costs 29s to promote, and keeps a second copy of the IR resident for the whole compile.
render_elixir_function_defs/3reads a module's whole IR out of the PLT, prunes it to the functions the page reaches, and encodes them. That happens once per page, per module, andencode_ir/2for a module definition is already a per-function fold, so the same function is encoded again for every page that reaches it.Measured on an app with 9,018 modules and 158 pages: 207,292 function encodings, 3,810 of them distinct. Every function is encoded about 54 times on average.
The reads cost more than the encodes. ETS copies on read, and pages reach about 4.5 functions per module, so a page pulls a whole module IR, up to 55MB, to keep a handful of functions and discard the rest. Over one compile that moves 213GB out of the PLT, and
create_page_entry_filesspends 131 seconds of a 201 second compile there, which works out to roughly memory bandwidth.Worth knowing for protocol modules:
maybe_prune_protocol_dispatcher_function_defs/3selects their function defs against the whole page's reachable set rather than the module's own MFAs, so which functions a page includes is page-dependent there even though the encoding of each one is not.Measurements
Same app, prototype keyed on
{module, kept mfas}rather than per function:create_page_entry_filesCache held 1,143 entries, 59,162 hits against 1,527 misses, about 45MB as refc binaries. The misses exceed the entry count because concurrent tasks miss the same key before either inserts.
That coarser key measured well here only because 830 of the 844 reachable modules produce one variant across all pages. It degrades on apps whose pages reach different fragments of a module, where a
{module, function, arity}key does not.For comparison, moving the IR PLT to
:persistent_termfor zero-copy reads getscreate_page_entry_filesto 50.5s, but needs+MIscsraised at VM start (the default 1GB literal carrier cannot hold a 2.6GB IR table), costs 29s to promote, and keeps a second copy of the IR resident for the whole compile.