> there are approximately five languages that are presumed to be acceptable for core applications in the base system: [...] Python
I don't know if you've tried to get someone else's Python running recently, but it has devolved into a disaster effectively requiring containers to accurately replicate the exact environment it was written in.
Core system applications should be binaries that run with absolutely minimal dependencies outside of default system-wide libraries. Heck, I would go as far as to say applications in the critical path to repairing a system (like apt) should be statically linked since we no longer live in a storage constrained world.
> I don't know if you've tried to get someone else's Python running recently, but it has devolved into a disaster effectively requiring containers to accurately replicate the exact environment it was written in.
Please show me a project where you believe you "effectively require containers" just to run the code, and I will do my best to refute that.
> since we no longer live in a storage constrained world.
I think you do care about the storage use if you're complaining about containers.
And I definitely care, on principle. It adds up.
For reasons I can only assume have to do with poorly configured CI, pip gets downloaded billions of times annually (https://pypistats.org/packages/pip), and I assume those files get unpacked and copied all the time since there would be no good reason to use uv to install pip. That's dozens of petabytes of disk I/O.
> Please show me a project where you believe you "effectively require containers" just to run the code
I guess GP meant "containers" broadly, including things like pipx, venv, or uv. Those are, effectively, required since PEP 668:
https://stackoverflow.com/questions/75608323/how-do-i-solve-...
> "containers" broadly, including things like pipx, venv, or uv.
This statement makes no sense. First off, those are three separate tools, which do entirely different things.
The sort of "container" you seem to have in mind is a virtual environment. The standard library `venv` module provides the base-line support to create them. But there is really hardly anything to them. The required components are literally a symlink to Python, a brief folder hierarchy, and a five-or-so-line config file. Pipx and uv are (among other things) managers for these environments (which manage them for different use cases; pipx is essentially an end-user tool).
Virtual environments are nowhere near a proper "container" in terms of either complexity or overhead. There are people out there effectively simulating a whole new OS installation (and more) just to run some code (granted this is often important for security reasons, since some of the code running might not be fully trusted). A virtual environment is... just a place to install dependencies (and they do after all have to go somewhere), and a scheme for selecting which of the dependencies on local storage should be visible to the current process (and for allowing the process to find them).
> This statement makes no sense. First off, those are three separate tools, which do entirely different things.
They are all various attempts at solving the same fundamental problem, which I broadly referred to as containerization (dependency isolation between applications). I avoided using the term "virtual environment" because I was not referring to venv exclusively.
and storage is a big deal for debian! It is basically the other player in embedded Linux other than rolling your own where a 4GB or smaller emmc is pretty common.
this is not normally a problem for a distro where the environment is well known
It is such a non problem it forced them to hack a „fuck you this Python is owned by the distribution not you“ message into pip requiring you to agree to „breaking your system“ to use it.
Of all the languages, python in the base system has been an unmitigated garbage fire.
> it forced them to hack a
It was not their action, nor is it hacked, nor is the message contained within pip.
The system works by pip voluntarily recognizing a marker file, the meaning of which was defined by https://peps.python.org/pep-0668/ — which was the joint effort of people representing multiple Linux distros, pip, and Python itself. (Many other tools ignore the system Python environment entirely, as mine will by default.)
Further, none of this causes containers to be necessary for installing ordinary projects.
Further, it is not a problem unique to Python. The distro simply can't package all the Python software out there available for download; it's completely fair that people who use the Python-native packaging system should be expected not to interfere with a system package manager that doesn't understand that system. Especially when the distro wants to create its tools in Python.
You only notice it with Python because distros aren't coming with JavaScript, Ruby etc. pre-installed in order to support the system.
Well the essential system Python should be in /usr/sbin and read-only (insofar Python allows that with its __pycache__ spam).
The fact that users have to keep up with multiple PEPs, error messages, --single-version-externally-managed, --break-system-packages, config files everywhere, stealth packages in .local and uv to paper over all of this shows that Python packaging is completely broken.
> the essential system Python should be in /usr/sbin
There's still quite a bit you can do with the "system Python". Mine includes NumPy, bindings for GTK, QT5 and QT6, Freetype, PIL....
> insofar Python allows that with its __pycache__ spam
This is, to my understanding, precisely why the standard library is pre-compiled during installation (when the process already has sudo rights, and can therefore create the `__pycache__` folders in those locations). This leverages the standard library `compileall` module — from the Makefile:
> The fact that users have to keep up with multiple PEPs, error messages, --single-version-externally-managed, --break-system-packages, config files everywhere, stealth packages in .local and uv to paper over all of this shows that Python packaging is completely broken.Please do not spread FUD.
They don't have to do any of that. All they have to do is make a virtual environment, which can have any name, and the creation of which is explicitly supported by the standard library. Further, reading the PEPs is completely irrelevant to end users. They only describe the motivation for changes like --break-system-packages. Developers may care about PEPs, but they can get a better summary of the necessary information from https://packaging.python.org ; and none of the problems there have anything to do with Linux system Python environments. The config files that developers care about are at the project root.
Today, on any Debian system, you can install an up-to-date user-level copy of yt-dlp (for example) like so, among many other options:
You only have to know how one of many options works, in order to get a working system.> All they have to do is make a virtual environment
Okay so to create a five line script I have to make a virtual environment. Then I have to activate and deactivate it whenever using it. And I have to remember to update the dependenceis regularly. For my five line script.
Seems to me the companies managing mloc-codebases pushed their tradeoffs on everyone else.
You, too: please do not spread FUD.
> Okay so to create a five line script... For my five line script.
I can guarantee that your "five line script" simply does not have the mess of dependencies you imagine it to have. I've had projects run thousands of lines using nothing but the standard library before.
> Then I have to activate and deactivate it whenever using it.
No, you do not. Activation scripts exist as an optional convenience because the original author of the third-party `virtualenv` liked that design. They just manipulate some environment variables, and normally the only relevant one is PATH. Which is to say, "activation" works by putting the environment's path to binaries at the front of the list. You can equally well just give the path to them explicitly. Or symlink them from somewhere more convenient for you (like pipx already does for you automatically).
> And I have to remember to update the dependenceis regularly.
No, you do not in general. No more so than for any other software.
Programs do not stop working because of the time elapsed since they were written. They stop working because the world around them changes. For many projects this is not a real concern. (Did you know there is tons of software out there that doesn't require an Internet connection to run? So it is automatically invulnerable to web sites changing their APIs, for example.) You don't have to remember to keep on top of that; when it stops working, you check if an update resolves the problem.
If your concern is with getting security updates (for free, applying to libraries you also got for free, all purely on the basis of the good will of others) for your dependencies, that is ultimately a consequence of your choice to have those dependencies. That's the same in every language that offers a "package ecosystem".
This also, er, has nothing to do with virtual environments.
> Seems to me the companies managing mloc-codebases pushed their tradeoffs on everyone else.
Not at all. They are the ones running into the biggest problems. They are the ones who have created, or leveraged, massive automation systems for containers, virtualization etc. — and probably some of it is grossly unnecessary, but they aren't putting in the time to think about the problem clearly.
And now we have a world where pip gets downloaded from PyPI literally billions of times a year.
Thank you! Exactly what I wanted to explain.
Yet, if I write a dockerfile, and need to use perl, system perl is fine.
If I need a python script, I have to arrange for all the RUN lines to live inside a virtual environment inside the container.
People are too harsh on this. It's not hard to install a version manager and set your primary python to that. Which is just good hygiene.
My understanding of the reasoning is that python-based system packages having dependencies managed through pip/whatever present a system stability risk. So they chose this more conservative route, as is their MO.
Honestly if there is one distribution to expect those kinds of shennanigans on it would be Debian. I don't know how anybody chooses to use that distro without adding a bunch of APT sources and a language version manager.
yes because then you're starting to use non-distro python packages. If you want to do that, use a virtualenv, there is no safe other way (even if there was no python in the base system) .
Yes, the distro people are strong believers in virtual environments as best practice - for you, not them.
There's a good reason for this. The average user has no idea and doesn't care what language some random distro-packaged program is written in. They want to be able to run ubxtool or gdal_calc or virt-manager or whatever without setting up a virtual environment. Python developers on the other hand should be adept at such things, should they choose to use a non-distro packaged version of something.
The tricky part is when "users" start using pip to install something because someone told them to.
This should become the official error message!
Can you expand? I'm honestly curious how it is not a problem in Pythons case
That asks GP to prove a negative. What problems have you encountered, and why do you believe them to be common?
It really, really isn't. I wish this would stop being repeated so much.