While I see the usefulness of lazy imports, it always seemed a bit backward to me for the importer to ask for lazy import, especially if you make it an import keyword rather than a Python flag. Instead I'd expect the modules to declare (and maybe enforce) that they don't have side effects, that way you know they can be lazily imported, and it opens the door for more optimizations, like declaring the module immutable. That links to the performance barrier of Python due to its dynamic nature as discussed in https://news.ycombinator.com/item?id=44809387
Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time.
> it always seemed a bit backward to me for the importer to ask for lazy import, especially if you make it an import keyword rather than a Python flag
Exactly this. There must be zero side effects at module import time, not just for load times, but because the order of such effects is 1) undefined, 2) heavily dependent on a import protocol implementation, and 3) poses safety and security nightmares that Python devs don't seem to care much about until bad things happen at the most inconvenient time possible.
> Of course that doesn't solve the overhead of finding the modules, but that could be optimized without lazy import, for example by having a way to pre-compute the module locations at install time.
1) opt for https://docs.python.org/3/reference/import.html#replacing-th...
2) pre-compute everything in CI by using a solution from (1) and doing universal toplevel import of the entire Python monorepo (safe, given no side effects).
3) This step can be used to scan all toplevel definitions too, to gather extra code meta useful for various dynamic dispatch at runtime without complex lookups. See for example: https://docs.pylonsproject.org/projects/venusian/en/latest/i...
3) put the result of (2) and (3) as a machine-readable dump, read by (1) as the alternative optimised loading branch.
4) deploy (3) together with your program.
For optimizing the module finding, using a custom import hook was indeed what I had in mind!