It is still early, but I find that this experiment makes little to no sense and it is barely useful.
The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
80%+ of effective testing is not in the testing framework but in the code architecture.
The author doesn't mention how the code is being architected and managed.
For what it is worth, I found that forcing agents on DI/hexagonal architecture and forcing a trivial coverage check is quite useful and produce overall good enough code with relative little effort
I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.
But what's the initial setup? Was it greenfields every time? And _how_ were the different testing frameworks used? What's the AGENTS.md there? So many things can influence these tests in positive/negative ways that are not included in the write up or results. Seems like a lot of effort without much in the way of actual helpful detail.
The setup is described in an earlier post linked in the second paragraph. Agents are given the zstd RFC and told to implement it, greenfield. Testing frameworks were used however the agents decided, which is to say, as mentioned repeatedly throughout the post, badly or not at all. Other factors that could conceivably affect the results can be assumed to be held constant, since the goal is to see what happens when you vary the instructions regarding which testing methodology should be used.
The setup is described in the linked post about programming language vs efficiency/token cost: https://danluu.com/pl-tokens/#zstd
> For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.
So basically, the agents were given the zstd RFC, told to implement a decoder, and also told to use a particular testing strategy (or not, for the control runs).
> The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure. Coupling to implementation details is unavoidable but should be managed, and ideally minimized, to keep the test suite robust and maintainable.
Given the choice between a test suite that monkey patches out some dependency (redis, say) on a per-test basis, and one which substitutes a fake for this dependency in some centralized composition root, which one do you prefer, and why?
You are not wrong but if we go deeper we will find more nuances.
> tests should test behaviour and not structure.
Of what?
The whole idea of software architecture, and underlying of my messages, is to structure the code so that it is easy, but before easy, possible, to work at the right abstraction level.
Which allows to tests the behaviour of components and not their structure.
Trivial example, say your code read from a socket and manage the bytes with some CPU operations and write them to another socket. (You may recognise it is basically what a compressor do)
The way in which you manage read and write to the sockets will make dramatically simpler or much more difficult to write good tests.
If you adopt strategies like sans-io, you will see that the testing is almost trivial.
If all the logic sprawl up from the read syscall in a loop, you will notice how more challenging testing becomes.
---
To answer your question, the way I let LLMs write code is very DI (dependency injection) based.
A class never instantiates another class - all the dependencies are passed to the constructor. Including time. Including whatever DB iteraction.
The reason why I prefer this is that I can test each component at every level of abstraction. I don't have to. But I can.
My current approach is to force coverage higher than say 80% as default and then when a bug is discovered drill down to the components.
The question "for which functions/modules/classes should I write tests?" and the question "how do I structure my functions/modules/classes?" are very closely related questions, which I think is what the other comment was getting at.
That follows largely from what you're saying though, so I think the two of you agree, you're just coming at the same thing from different perspectives. Like you say, you want to avoid/manage coupling tests to implementation details, and typically the best way to do that is to bundle a significant, testable chunk of code into a single module, define a clear public interface to that module, and then test that interface. The internals of that module are then free to change, but the test interface should stay the same.
But the corollary of that is that you can't just design your modules independently of your tests, you need to design them so that they are testable. Which is why the previous comment links testing to code architecture/design.
Regarding dependencies like that, the best situation is where you can either:
(a) design a module so that the dependency is injected with a clear interface (not just "here's an instance of libredis.rs" but "here's a series of callbacks for saving data, those callbacks might save to Redis, but they might be in-memory only, or they might save everything to the blockchain, either way it doesn't matter").
(b) just spin up an instance of Redis and test directly against that — it should be quick enough, and there's plenty of ways to ensure that the tests don't affect each other even while accessing a shared resource.