Am I the only one to stick with the std lib, read the docs and changelogs carefully, and implement functions I really need the way my application makes use of them?
I learned the hard way, that dependencies kill projects.
Not saying this isn't great, thanks for creating it! It does have its use cases, of course.
> Am I the only one to stick with the std lib, read the docs and changelogs carefully
I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time.
I don't want footguns in my code, I don't want code I have to write and test myself, and I don't want to have to become an expert in a domain before I can write something that serves my purpose.
For the datetime library, specifically, I'm switching to whenever for everything, because I've been bitten by conversions and naive/aware datetime confusion too many times.
My hope is that a lib like this one or similar could rally mindshare and become integrated as the new standard, and adopted by the wider developer community. In near term, it comes down to trade-offs. I see no decision that works for all use cases. Dependencies introduce ticking time bombs, stdlibs should be correct and intuitive, but at least when not they are usually well tested and maintained, but when stdlib don't meet urgent production needs you have to do something.
Link to Tom Scott & Computerphile from 10y ago on tz madness. https://www.youtube.com/watch?v=-5wpm-gesOY
Is there any part of the Python standard library written in Rust? I would see that as a big impediment to having Whenever adopted as standard.
I don’t think so, but they do offer a pure-Python version as well.
https://whenever.readthedocs.io/en/latest/faq.html#how-can-i...
True, but does it suffer in efficiency? I wish they had included the pure Python version in their benchmark.
It's basically what happened in Java. Everyone used jodatime, and they took great inspiration from that when making the new standard time api for java 8.
> I work in healthcare. If I have a choice between "reading docs/changelogs carefully, implementing functions", and "adding an extra dependency", I'm taking the dependency every single time.
This kinda sums up the sorry state of software engineering. People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI.
Taking on a dependency does not relieve you of reading docs, but it also adds a further burden as you now need to trust the code. The stdlib is much more heavily tested and documented than any 3rd party library will be.
The stdlib datetime module is in more of a sorry state than certain third party libraries and is full of footguns, as you, a read-the-docs-and-changelogs-carefully person, can surely tell from the myriad deprecations and warnings added over the years, not to mention the confusing params like tz and tzinfo. Heavily tested doesn’t mean shit when the semantics of the API is fundamentally flawed and can’t be updated.
> Taking on a dependency does not relieve you of reading docs, but it also adds a further burden as you now need to trust the code. The stdlib is much more heavily tested and documented than any 3rd party library will be.
Sure, but the opposite applies as well. Sticking with the flawed stdlib means you are trusting that every single future developer is as careful in reading all the docs as you are - even when it's someone reviewing that random trivial-looking patch which is secretly hiding a massive footgun. A junior developer submitted a five-line patch touching datetime? Better schedule several hours for a proper analysis!
Or you can write your own wrapper code, of course. Which will almost certainly have worse documentation and testing than a popular third-party library.
External libraries aren't evil. When chosen properly, they relieve you of burdens. You shouldn't grab any random "leftpad"-like dependency like they are going out of fashion, but something as complicated as timezone handling is best left to code written by domain experts - which means using a library.
> The stdlib is much more heavily tested and documented than any 3rd party library will be.
You initially said you write your own code instead of using libraries, I replied to that, and now it's that you use the stdlib instead of libraries. I won't argue against shifting goalposts.
The poster who you just replied to is not the same person who started the thread. I am yet another person.
In addition, the original post begins with, "Am I the only one to stick with the std lib". The goalposts are stable.
Again, the library in the article removes some stdlib footguns. The footguns are there, and if you use the stdlib, you need to remove (or avoid) them yourself. There's no free lunch, and you'll either need to remove them yourself (and test that code), avoid them (and constantly remember to not hit them), or use another library.
It's not a choice between "using a dependency" or "using something in the stdlib", where all other code remains the same, otherwise there would be no point to writing a library, as it would offer nothing over `datetime`.
That’s why I’m a software developer though, because I learned the footguns in the standard library. I use it before third party libraries whenever I can. I can’t understand the fear, just learn your job.
If your approach to footguns is "just learn your job", I'm sorry, but you aren't a very good software developer.
Well you know nothing about me so screw off.
You felt you knew enough about me to tell me to learn my job, so I figured you'd be open to similar feedback.
True, my bad. I apologize. I’m impressed by your grace with this comment.
Did OP blindly install the package?
Or do it with sight?
> People can't even be bothered to read docs but will just blindly install a package just because someone was able to package it and upload it to PyPI.
That's a straw man argument. No one said "blindly". You can very well carefully consider the pros and cons of adding a dependency and arrive at the conclusion that it makes sense. Many PyPI packages are in the Debian stable repositories, you could use that as an additional barrier as well.
To be fair, comment^^ actually said
which is what comment^ answers to, which to me actually sounds like that the added dependency comes in place of "reading docs/changelogs carefully".I think it matters a lot how much one can trust a 3rd party library, how much it is used, how much it is maintained etc. Moreover, it also matters how central and important this is to what you are actually doing, for example if the datetimes I come across are pretty much specific and with limited scope, I would probably care about reading docs less than if I am reading data with datetimes that may be totally wild. Furthermore, there are some libraries that are just considered the standard in some fields. If I use python I call pandas to read csvs, I am obviously not gonna write my own csv parser. It will also make your code more readable for others that already know and use the same libraries if they are so standard. But that's probably a very small minority of libraries that people actually use in certain languages.
> If I use python I call pandas to read csvs
So it's you that isn't just using the built in csv parser in this project I inherited. Come back and repent.
How could it possibly be true that libraries that are so commonly used that you consider them “the standard” are so infrequently used as to be a “very small minority” of libraries people actually use?
I am not sure I understand the comment. I used pandas as an example of sth I frequently encounter when dealing with python code dealing with data frames. Is it not commonly used? What’s your argument here, I am genuinely asking.
I'm trying to square two aspects of your comment:
> Some libraries are so common that basically everybody uses them, to the point that they're considered the standard and they don't count when I raise complaints about people pulling in dependencies
> "But that's probably a very small minority of libraries that people actually use in certain languages."
It seems like by definition, the majority of libraries that people are actually using are going to be the popular ones that everybody is using.
I get where you’re coming from. There’s a price you pay though eventually. You’ll have to thoroughly vet all your dependencies for malicious code at some point. Otherwise how do you have any clue what you’re running?
There are so many footguns in the datetime lib.
That's why I use a Flake8 plugin to prohibit especially egregious footguns.
https://github.com/jkittner/flake8-ban-utcnow
Honestly yeah who in tarnation created that function and called it utcnow
These things are really frustrating
Yeah, utcnow is completely broken. They should have fixed it when they created datetime.timezone.utc, but they didn't. The recommendation is to use datetime.datetime.now(datetime.timezone.utc) instead. utcnow should be deprecated and eventually removed.
I recently learned that there is datetime.UTC which is an alias of datetime.timezone.utc
It helps a bit with the verbosity:
from datetime import datetime, UTC
datetime.now(UTC)
That's from 3.11 onwards.
You are a sad minority, IME. I’m right there with you. I extended the uuid library to generate UUIDv7, based off of the RFC. It’s pretty easy to implement, as it turns out. Overruled, because “we don’t want to have to maintain additional code.” As if the ABI for bitshifts is going to change?!
There’s an out of sight, out of mind mentality with dependencies.
As long as there is a conscious decision to build or ‘buy’, it’s fine. I think some people can be a little too careless with adding dependencies though, not realising they can have an equal if not greater maintenance burden.
this is a great idea if you want to slow down your project. most projects start with few rules and “best practices” like this. everyone is free to pull in dependencies as needed. because they are needed. but then once the project grows larger, those who have been around longer want to reverse course and gatekeep dependencies. but this is the opposite of what helped the project grow initially. and later contributors have a harder time making similar progress because they have to fight to add basic libraries. ensuring that efficiency per engineer goes down
I think this is fairly unrealistic. Does all your datetime manipulation involve proper use of the fold parameter as indicated in the article?
Are you saying you never pull in dependencies? Why stop there, why not re-implement the std lib as well? Surely there is a sensible middle ground: If you only need a small part of a dependency, consider implementing it. If you make heavy use of a dependency and want to benefit of years if not decades of dedicated developers testing and maturing its code, with a large community who has already stepped in all pitfalls you might step into and collectively encountered all the edge cases, just use the dependency.
Creating from scratch also creates hidden debt, it's just moved onto yourself. Especially when working with dates and timezones.
I cannot imagine having the spare time to invest in building date/time foundations and maintaining them through changes to DST timing and country/time zone changes.
The only crazier idea I can think of is implementing character encoding conversions myself.
Seriously.
Nobody needs a package for "left-pad", which is the most infamous example.
But there are a lot of areas where it's not a good use of your time to reinvent the wheel. There's no moral virtue in writing everything yourself. And popular packages have gone through a more bug-finding and bug-fixing than your personal code probably ever will.
A library that goes "poof" when you need to upgrade it is also a hidden debt
There is also a distinction to be made between "technical" and "political" dependencies. Technical dependencies usually track some spec or common consensus, and political dependencies are e.g. timedate-library. Political dependencies are almost like business logic, because they have to track changing political decision made all over the world (or be local to some country).
Timedates are hard, and units may require even harder historical/present "political" tracking as how they are defined, and I would never want to maintain this kind of dependency: https://github.com/ryantenney/gnu-units/blob/master/units.da...
And what comes to timedate problems, I try to keep it simple if the project allows: store and operate with UTC timestamps everywhere and only temporarily convert to to local time (DSTs applied, if such) when displaying it in user-facing UI. This functionality/understanding can be locked into own 20-line microlibrary-dependency, which forces its responsible person to understand country's timezone and when e.g. DST changes and where/how/who decides DST changes and what APIs is used to get UTC time (and of course, its dependencies, e.g. NTP, and its dependencies. e.g. unperturbed ground-state hyperfine transition frequency of the caesium-133 atom, which result is then combined with the Pope Gregory XIII's Gregorian calendar, which is a type of solar calendar mixed with religious event for fixing the time, which which is then finally corrected by rewinding/forwarding "the clock" because its too bright or dark for the politicians).
A service goes poof, a library either slowly deteriorates or breaks just as easily as a self-written one (if an underlying platform breaks it for some reason). The self-written one is maintained by 1 person, the other is used by 100+ people who could jump in a collaborate on its fixing.
I would still rather using a library for dates, a million times so.
> a library either slowly deteriorates or breaks just as easily as a self-written one
Yes, I agree with this
> The self-written one is maintained by 1 person, the other is used by 100+ people who could jump in a collaborate on its fixing.
Libraries that have 100 people collaborating on it are very few
Most likely you'll have to vendor it and fix whatever issues you have.
Even worse when it's a dependency of a dependency and you also use it, so, let's say a dependency of yours bumps the version but this breaks your code. (Not sure this breaks only in python or in js as well, but it's possibly that it does)
Honest question: why is it so common for software developers to not upgrade their dependencies on regular intervals?
I can’t for the life of me figure out why. If you update everything incrementally you bypass the upgrade version problem when you’re so far behind that so much has changed that it becomes an overwhelming burden.
I think frozen dependencies are a big anti pattern, and places where I work that regularly update their deps tended to have better software practices generally
Because time spent updating dependencies won't contribute to adding new features.
Besides, any update risks breaking stuff. Not freezing dependencies isn't an option, because that means any commit can cause breakage in a completely unrelated part of the codebase, in a way which can be extremely confusing to debug. And you don't really want to install the very newest versions either, better wait a week or two for someone else to run into bugs and release a .1 version.
The sweet spot is somewhere in the middle: update often enough to avoid updates becoming a massive burden, stick with fixed versions between updates. I reckon it's best to just schedule some dedicated time for it every month or two.
It typically takes me maybe an hour to update my dependencies? I run type checks and e2e tests as part of that to have a relatively high degree of confidence that nothing has been broken. Also splitting the change into multiple steps (i.e. do minor/patch upgrades first, check nothing's broken, run a major upgrade, check nothing's broken, etc) means it's fairly to see where something is causing problems and needs to be handled more carefully.
I do this typically every couple of weeks, and it takes up almost no time at all in comparison to time spent on other work. Someone needs to review the eventual PR created, but that's also typically fairly easy. NPM makes this all very easy to do. In Python I've used tools like PDM or uv to handle dependencies similarly.
Yes, in python only with modern tools this is somewhat feasible
And you still have upgrades that break interfaces and such
That’s why we do it on an interval and not when it’s first available. Avoids bleeding edge bugs but prevents the overwhelming factor of being years out of date. It typically gets handed out at the end of the month or there about.
Being judicious in selecting dependencies goes a long way too. Not always easy but certainly worth the time
I'm glad to hear that all libraries you use honor semver religiously. For the rest of us stuck here on planet Earth it's rolling the dice what, exactly, is going to happen going from 1.7.12 to 1.7.13 for any random dep. The only way to find out is to try it and then unfuck things when it spits out some transitive error message because it turns out that package-A and package-F hate each other but only on Thursdays with a full moon
They don’t all follow semver perfectly or in some cases at all, but we have an interval when we do upgrades.
Our test suite is comprehensive and will catch most breakages automatically. The key to success is robust testing, as it cuts the manual footprint significantly.
This does mean we are quite judicious with selecting dependencies.
It isn’t all that complicated when everyone is following best practices most of the time I have found[0]
It still leavings me wondering in a lot of cases
[0]: perhaps this is the real heart of the issue is best practices are systematically ignored. I’ve worked at places like that and it’s no wonder they grind through folks
I mean this seriously: congratulations on working someplace that has such rigorous requirements that you are able to write a comprehensive test suite that doesn't get completely rewritten every month or so. I guess because I enjoy working in startups, the cost I have to pay is that "requirements" are spelt "fever dream" and the "acceptance criteria" field in Jira is often just "." (because it's a required field, donchano)
A member of staff who goes “poof” as the only one who understands your wrapper library that has a critical bug is a more common kind of hidden debt.
Functions that you have to document, test and maintain of course. You do that, right? And all the people in your team, they do that and will keep doing that once you leave, right? And they all understand the business domain and all the pitfalls that come with it and have the skill, time, and resources to take care of it, right?
And this for every single problem: time, text, maths, network, parsing, formatting, validating, authenticating...
Curious about examples of projects being killed by dependencies.
While I've never seen a project killed by dependencies, I've certainly seen projects stuck on treadmill of constant dependency updates.
You know, they import 5 libraries, each of which imports 5 more libraries, each of which imports 5 more libraries, and suddenly they're buried in 'critical' updates because there's a denial-of-service bug in the date parser used by the yaml parser used by the configuration library used by the logging library used by the application.
Not killed IME but bloated and dragged down by tech debt.
E.g the JS project that uses the stdlib Date API, and pulls in moment.js, and also uses date-fns.
Or the one that pulls in bits and pieces of lodash, ramda, and other functional libraries.
And maybe it uses native fetch and axios depending on the current phase of the moon.
They don’t die but time is wasted in code review trying to understand if there is any kind of deliberate approach behind the scattershot application of packages with duplicated purposes.
(picking on JS is perhaps unfair but it’s probably the most egregious example of dependency hell)
The Date example is possibly an even better example of why dependencies that get hard problems right are so important. If Python's datetime library is bad, Date is truly terrible. Every time I have used it I have regretted it long term (including when using it in combination with date-fns in the hope that that makes it usable). And in the end, trying to keep things simple with Date has caused more technical debt than using a sensible dependency.
Some problems simply require using the right tools. They aren't necessarily hard, but they will be if you try to hammer a nail in with a screwdriver. The Date API, and to a certain extent Python's datetime module, are screwdrivers for a nail-shaped problem.
The rest of your example seem to have more to do with bad dependency practices than using dependencies in the first place. If you are going to include a dependency, think about it, consider whether it's worth it, document that decision, and then consistently use that dependency. Just because you've seen projects use dependencies poorly doesn't mean dependents are bad by themselves.
I used a bad example with the Date API, except that, say, using moment or date-fns and sticking to it would be better than jumbling them up. If you’re going to do a migration from one to the other then it has to be committed to.
Dependencies themselves can’t be good or bad. They only acquire that property in the context of the dependent codebase. And a codebase that you can’t spin up from scratch or easily grok because of its approach (or lack thereof) to dependencies is almost certainly on the bad end of the spectrum.
I’ve seen this destroy some python codebases internally. In fact, my choice of a JS example was unfair because it becomes an absolute shitshow in python once you have dependencies and transitive dependencies with very tight version constraints.
Fortunately, JS is getting Temporal, which will be great. No such thing on the horizon for Python yet.
Well, there's libraries like this one, which is the point here.
And even in JS, Temporal won't be available broadly for a good while yet (it will probably be rolling out in Firefox in a couple of months' time, but in Safari it's still behind a feature flag, and I don't think there's even a feature-flagged implementation for Chrome yet). In the meantime, it makes sense to use a polyfill — again a library.
By all means choose your dependencies wisely, but the point I'm trying to make is that very often a sensible use of dependencies will reduce your technical debt, and attempting to use bad approaches to complex topics just because they're built into the standard library will cause so many problems down the line.
I fully agree with that. Just a bit disappointed that Python is not even considering to fix/replace it's problematic datetime library (as far as I know). Excellent third-party library is good, but an excellent standard library is even better!
Maybe this is the first step toward that future.
In Temporal's case, it takes significant inspiration from several existing datetime libraries in Javascript. Those were all stepping stones (among others) that led to Temporal.
I'm not the creator, the credit for that goes to Arie Bovenberg. I just wanted to show this to people.
As others stated, there are many rough edges and footguns in the stdlib. BUT ... in my (and yours apparently) opinion, it's a matter of knowing those edges/guns, and work with them. Like you, I also prefer to create my own code around those instead of bringing in some library that brings in their own foot guns and possibly sub-dependencies and and and...
So your projects end up with their own "lib" of scattered time functions, possibly with new small bugs. I'd then rather have a proper well-tested and maintained library.
The "lib" one needs for date time functions is as good as non-existent.
But yeah, you will have to create tests for the codepath, instead of relying on the tests the library maintainers create.
Given the list of issues (most of which I had never heard of) that the library author describes with datetime, I think I trust his tests over the ones I’d write.
Given how many tests I see fail on 29th February at the companies I’ve been at, I don’t trust my colleagues’ tests either!
In a previous job where we had complete hell with an internal datetime lib under ambiguous ownership, a colleague joked that the original authors had taken early retirement when they turned 2,147,483,647 years old.