It's... really the same problem when you hire people to just write tests. A lot of time it just confirms that the code does what the code does. Having clear specs of what the code should do make things better and clearer.
It's... really the same problem when you hire people to just write tests. A lot of time it just confirms that the code does what the code does. Having clear specs of what the code should do make things better and clearer.
Yep, tests written after the fact are just verifying tautologies.
> Most teams don't [write tests first] because thinking through what the code should do before writing it takes time they don't have.
It's astonishing to me how much our industry repeats the same mistakes over and over. This doesn't seem like what other engineering disciplines do. Or is this just me not knowing what it looks like behind the curtain of those fields?
When push comes to shove, software can usually be fudged. Unlike a building or a water treatment plant where the first fuck up could mean that people die.
I like to think that people writing actual mission critical software try their absolute best to get it right before shipping and that the rest our industry exists in a totally separate world where a bug in the code is just actually not that big of a deal. Yeah, it might be expensive to fix, but usually it can be reverted or patched with only an inconvenience to the user and to the business.
It’s like the fines that multinational companies pay when breaking the law. If it’s a cost of doing business, it’s baked into the price of the product.
You see this also in other industries. OSHA violations on a residential construction site? I bet you can find a dozen if you really care to look. But 99% of the time, there are no consequences big enough for people to care so nobody wears their PPE because it “slows them down” or “makes them less nimble”. Sound familiar?
Quite. We’re far more similar to construction workers than we are civil engineers, despite the lofty title we like to bestow upon ourselves.
And from slightly different view. What we make is not output of modern mass production. With highly tuned and most of time perfectly matching parts build into one unit.
Instead we make pre-mass production bespoke products where each part is slightly filled and fitted together from bunch of random components. Say the barrel can't be changed between two different handguns. We just have magic technology to replicate the single gun multiple times. Does not mean it is actually mass-produced in sense say our current power tools are.
Software is unusually forgiving about skipping process because the worst outcome is usually a rollback, not physical damage or liability. In electronic and civil engineering the cost of mistakes is much higher so up-front planning is the norm. If a bug in code bricked production hardware overnight you'd see test-driven practices everywhere.
That's probably dependent on your specific area of work. For most projects, It's okayish to deploy code with bugs. There will be future releases that fix those bugs and add improvements. Obviously that's not the case with high risk systems like space rockets software and similar.
With other engineering professions, all projects are like that. You cannot "deploy a bridge to production" to see what happens and fix it after a few have died
a lot of the value of tests is confirming that the system hasn't regressed beyond the behavior at the original release. It's bad if the original release is wrong, but a separate issue is if the system later accidentally stops behaving the way it did originally.
The issue I see is that the high test coverage created by having LLMs write tests results in almost all non-trivial changes breaking tests, even if they don't change behavior in ways that are visible from the outside. In one project I work, we require 100% test coverage, so people just have LLMs write tons of tests, and now every change I make to the code base always breaks tests.
So now people just ignore broken tests.
> Claude, please implement this feature.
> Claude, please fix the tests.
The only thing we've gained from this is that we can brag about test coverage.
My best unit tests are 3 lines, one of them whitespace, and they assert one single thing that's in the requirements.
These are the only tests I've witnessed people delete outright when the requirements change. Anything more complex than this, they'll worry that there's some secondary assertion being implied by a test so they can't just delete it.
Which, really is just experience telling them that the code smells they see in the tests are actually part of the test.
meanwhile:
is demonstrably a dead test when the story is, "allow users to have multiple shipping addresses", as is a test that makes sure balances can't go negative when we decide to allow a 5 day grace period on account balances. But if it's just one of six asserts in the same massive tests, then people get nervous and start losing time.Unit tests vs acceptance tests. You shouldn't be afraid to throw away unit tests if the implementation changes, and acceptance tests should verify behavior at API boundaries, ignoring implementation details.
BDD helps with this as it can allow you to get the setup out of the tests making it even cheaper for someone to yeet a defunct test.
I feel it end up a massive drag on development velocity and makes refactoring to simpler designs incredibly painful.
But hey, we're just supposed to let the AIs run wild and rewrite everything every change so maybe that's a heretic view.
>simpler designs
Some complex design might just be hacks on hacks, but some are chesterton's fences
Yeah. The purpose of tests is to verify correctness of their setUp mocks :)
yup agree - i think have specs and then do verifications against the spec. I have heard that this is how a lot of consulting firms work - you have acceptance criterias and thats how work is validated.