> not mostly the test, but the real world production scars
Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour.
SQLite is a good example to bring up because its extensive closed-source tests are what’s often cited as being what keeps people from forking it. (Turso did it, though, but it takes a company to deliver some guarantee of equivalent diligence.)
And yes, years and years of running.
Sure, but behaviors that never have a bug or regression don't get a test. Software of this kind of complexity has all kinds of behavior that has never been broken, and doesn't have a specific test written for it.
Getting an extensive test suite passing is certainly orders of magnitude better than having no test suite at all, but it still doesn't tell you as much as you need to know. I would absolutely never trust an LLM Postgres rewrite (in any language) in production based on "only" Postgres's test suite passing.
> Software of this kind of complexity has all kinds of behavior that has never been broken
This space of things is astronomically larger than the space of things expressly covered by any test suite.
"Program testing can be used to show the presence of bugs, but never to show their absence." -Edsger W. Dijkstra
I've also seen situations where a customer reports a bug, the fix breaks some regression, and the updated behavior to work around the fix breaking the regressions turns into an undocumented feature.
How do you break a regression? A regression is breakage. Are you one of the people who use "regression" to mean "regression test"? Did Codex learn this from you? I hate it.
Yeah, sorry, it was a codebase with _only_ regression tests dating to the 80s, so we called everything a "regression".
The same basically holds for proofs in the absence of coherent global correctness criteria like, say, confluence and normalization for a lambda calculus, or soundness and completeness for a logic.
Fable's napkin estimate of the effort required to produce a passable reference semantics for Postgres, which would involve novel discoveries in denotational semantics of concurrent transactions and so on, might be in the ballpark of 30–60 years of PhD level work.
So realistically I think the only way to validate a Postgres implementation involves differential testing, fuzzing, acceptance test suites, etc. And still you'll have bugs that need to be hammered out the good old fashioned way.
Or even a human rewrite merely because some language is the current fad. A rewrite in a different language should be done for very good reasons, to solve problems that are bigger than the costs of all the bugs that will be introduced.
Perhaps before embarking on one of these rewrites the first step should be a heavy round of mutation testing and property based testing. Contribute any new testing code from this back to the original project. And *then* embark on the rewrite.
If that's your concern, then your argument becomes "software should never change". Why dare patch any bug ever? It might be load-bearing in some unknown, undocumented, unsupported workflow somewhere in the world. No test imaginable can catch that apart from the scream test.
There are reasonable arguments against language ports, but this is not one. You're making an argument against code changing at all ever.
Agreed.And a rewrite in another language creates a high probability of a change in behaviour
The maintainers that wrote those tests will have experience you won't get out of a rewrite.
I think this is also where the real work is. A rewrite is one thing, that you can show off with a flashy blogpost. The maintenance, for years to come, won't be of that nature yet it still requires as much work.
This feels like the image of the plane that returns from battle with bullet holes, and the engineer being asked to path up where the holes to make it stronger. Only to be told to patch where there weren't holes as those planes didn't make it home.
While not an exact fit of an analogy, those tests patch what was a problem with Postgres in the wild. What it doesn't cover are the things that worked in Postgres without tests, but may fail in port and go undetected.
I don't necessarily disagree, but two other points to consider:
1. Every test that is written is another use case that wasn't tested before. 100% test coverage is often impractical, but the more tests you have the more of the code you can be confident about.
2. Every test you add is another regression that can't happen in the future; if you test the index rebuilding code and validate the output then you know that you aren't going to make a change that breaks the index rebuilding code. If you have a legitimate change you update the tests, but if you're not expecting the change then you know there's a bug somewhere.
Anything that doesn't have tests is unspecified behaviour. While it is true that a port may differ in behaviour where the behaviour is unspecified, "fail" is not the right framing as there is no definition of what it should do.
sqlite is the pathological case though; it has ~590 times more SLOC in the test suite than in the actual sqlite project.
https://sqlite.org/testing.html
> Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour.
If you can be 100% guaranteed that there indeed is a test for every occurred bug. Sometimes maintainers are not so strict about it.
And some programmers are so good that some issues are self-explanatory and they write good code to note a thing but don't write a test, because implementing the test is more expensive.
> And some programmers are so good that some issues are self-explanatory and they write good code to note a thing but don't write a test, because implementing the test is more expensive.
You don't write a test (just) to verify that your change fixed the issue, but to ensure it doesn't regress in the future after an unrelated refactor.
a code written to pass a test can surface unintended new bugs.
very naive. the runtime behavior of a rewrite should be significantly different in all kinds of unpredictable ways nobody see coming or might expect. It is a combination of language semantics, compiler behavior, operating system behavior, file system behavior, driver behavior, ..
yea, I found LLMs poor at reasoning through system-wide behaviors. Good luck to the vibers when they try to rootcause a rare data-corruption problem. They will hand it off to fable which will attempt trial-and-error bug fixes which will eventually muck up the code. What happens then ?
One issue is those are the bugs you get when you write it in C++.
They aren't the bugs you get when you write it in Rust.
The kind of bugs you get are usually a function of the problem, language, implementation approach.
So you get other bugs when rewriting in another language without existing tests, got it. This is why I hate all the announcements of "it is rewritten in rust so it is obviously better than the original since it passes all the tests". Edit: and it's an LLM rewrite. Add that to the pile of over hyped messaging.
Unfortunately, too many people are getting captured by marketing and are divorcing themselves from reality. A rewrite can be an improvement, even if in the same or any other language.
But, there are also levels, in terms of quality and human code review, when dealing with rewrites. New bugs can be introduced or there can be style issues, that can take time to fully reveal themselves, and particularly if the person or people involved are not familiar with the other language.
having a large test suite does not equal to testing every potential edge scenario that has never broken before in production.
So many comments here talking about the downsides. The only reason to do a rewrite is because there are massive upsides. Maybe the implicit point is that the upside (memory safety must be the biggest), isn't worth the downside (lots of bugs to be figured out before you trust it).
I find a lot of HN discussions quickly turn into thought experiments and philosophical debates that largely forget the original topic. For the most part, I find this idiosyncrasy charming and entertaining, but it does frequently result in forests being missed for the trees.