new | ask | show | jobs
btown 3 days ago  [ - ]

Has anyone had success with specific prompts to avoid the agent over-indexing on implementation details? For instance, something like: "Before each test case, add a comment justifying the business case for every assumption made here, without regards to implementation details. If this cannot be made succinct, or if there is ambiguity in the business case, the test case should not be generated."

freedomben 3 days ago  [ - ]

I've had reasonable success from doing something like this, though it is my current opinion that it's better to write the first few tests yourself to establish a clear pattern and approach. However, if you don't care that much (which is common with side projects):

Starting point: small-ish codebase, no tests at all:

    > I'd like to add a test suite to this project.  It should follow language best practices.  It should use standard tooling as much as possible.  It should focus on testing real code, not on mocking/stubbing, though mocking/stubbing is ok for things like third party services and parts of the code base that can't reasonably run in a test environment.  What are some design options we could do? Don't write any code yet, present me the best of the options and let me guide you.
    > Ok, I like option number two.  Put the basic framework in place and write a couple of dummy tests.
    > Great, let's go ahead and write some real tests for module X.
and etc. For a project with an existing and mature test suite, it's much easier:

    > I'd like to add a test (or improve a test) for module X.  Use the existing helpers and if you find yourself needing new helpers, ask me about the approach before implementing
I've also found it helpful to put things in AGENTS.md or CLAUDE.md about tests and my preferences, such as:

    - Tests should not rely on sleep to avoid timing issues.  If there is a timing issue, present me with options and let me guide you
    - Tests should not follow an extreme DRY pattern, favor human readability over absolute DRYness
    - Tests should focus on testing real code, not on mocking/stubbing, though mocking/stubbing is ok for things like third party services and parts of the code base that can't reasonably run in a test environment.
    - Tests should not make assumptions about the current running state of the environment, nor should they do anything that isn't cleaned up before completing the test to avoid polluting future tests
I do want to stress that every project and framework is different and has different needs. As you discover the AI doing something you don't like, add it to the prompts or the AGENTS.md/CLAUDE.md. Eventually it will get pretty decent, though never blindly trust it because a butterfly flapping it's wings in Canada sometimes causes it to do unexpected things.