you are less likely to have the same vulnerabilities as everyone else though, so unless somebody is targeting you, those vulnerabilities might not matter all that much
I mean... you can literally do that now. You can set up a loop to iteratively pentest, review and patch a codebase (with human supervision as you prefer) and it'll find and fix more vulnerabilities in a day than a pentest team used to find in a quarter, for a tiny fraction of the price.
This isn't a joke, this is now part of my pre-launch SOP. I even have it tracking everything so I can log stuff to fix vs. known shippables vs intentional design/false positives vs. upstream stuff which doesn't have a fix available yet, and keep track of which builds have the fixes. Almost entirely automated, I mostly review the findings and do some categorization/enrichment during the pentest review stage, and do a human code review pass as patches are submitted.
Stuff that used to take me multiple hours to write a fix for and then weeks to get code reviewed and deployed now get done in minutes.
How do you actually do that? Is it all running locally? Cloud agents? Would love to hear about this. I see these deep agent loops mostly just burning tokens, but when I guide the AI I get very good results, so I’m not sure where the disconnect lies.
I use Zed (https://zed.dev) as my agent harness and either the $20 ChatGPT sub + Sol for personal/independent projects or an enterprise Claude account for sponsored/paid work. From the stats for my current work I use about $400/mo in tokens and a lot of that is non-coding work like pruning JIRA, managing business documentation, making dashboards - so my true coding agent cost is significantly less.
It's pretty simple, you could probably set up something like that by:
Configure some kind of CLI tool to talk to your ticketing system and git repo so you can programmatically interact with them. If you don't have a ticketing system, instruct the agent to use local text or markdown files to track issues and progress.
Ideally, make your code runnable in a way the agent can use. For my webapps I build a test harness so that I can run all the endpoints and workflows via reproducible tests against an embedded database. This is easier than it sounds, e.g. there are libraries out there to embed PostgreSQL or SQLite into source code, you can set up a test harness so you can run unit tests, integration tests and workflow tests that use your real frontend, server and database.
Paste this comment thread into the agent prompt and tell it to run a similar loop on your code base: a session that searches for vulns and writes up a report, some way for a human to do a review pass on the report, a session that indexes the reviewed findings into tickets, and sessions that fix the fixable issues and submit patches to your repo. The next search session should first read all the open issues so it doesn't duplicate work of earlier sessions.
LESS IS MORE - avoid fancy agent tooling and skills, don't cargo cult from others, build your own tools as you find your own needs. If something can be automated, use the agent to write tools and tests for it, don't just keep prodding the agent to do it.
you are less likely to have the same vulnerabilities as everyone else though, so unless somebody is targeting you, those vulnerabilities might not matter all that much
Because LLMs output is always so unique and original. That's why it looks all the same right?
Wot? You are incredibly likely to have the same vulnerabilities as everyone else (as they exist in your LLM’s training corpus).
So security through obscurity.
I mean... you can literally do that now. You can set up a loop to iteratively pentest, review and patch a codebase (with human supervision as you prefer) and it'll find and fix more vulnerabilities in a day than a pentest team used to find in a quarter, for a tiny fraction of the price.
This isn't a joke, this is now part of my pre-launch SOP. I even have it tracking everything so I can log stuff to fix vs. known shippables vs intentional design/false positives vs. upstream stuff which doesn't have a fix available yet, and keep track of which builds have the fixes. Almost entirely automated, I mostly review the findings and do some categorization/enrichment during the pentest review stage, and do a human code review pass as patches are submitted.
Stuff that used to take me multiple hours to write a fix for and then weeks to get code reviewed and deployed now get done in minutes.
How do you actually do that? Is it all running locally? Cloud agents? Would love to hear about this. I see these deep agent loops mostly just burning tokens, but when I guide the AI I get very good results, so I’m not sure where the disconnect lies.
I use Zed (https://zed.dev) as my agent harness and either the $20 ChatGPT sub + Sol for personal/independent projects or an enterprise Claude account for sponsored/paid work. From the stats for my current work I use about $400/mo in tokens and a lot of that is non-coding work like pruning JIRA, managing business documentation, making dashboards - so my true coding agent cost is significantly less.
It's pretty simple, you could probably set up something like that by:
Configure some kind of CLI tool to talk to your ticketing system and git repo so you can programmatically interact with them. If you don't have a ticketing system, instruct the agent to use local text or markdown files to track issues and progress.
Ideally, make your code runnable in a way the agent can use. For my webapps I build a test harness so that I can run all the endpoints and workflows via reproducible tests against an embedded database. This is easier than it sounds, e.g. there are libraries out there to embed PostgreSQL or SQLite into source code, you can set up a test harness so you can run unit tests, integration tests and workflow tests that use your real frontend, server and database.
Paste this comment thread into the agent prompt and tell it to run a similar loop on your code base: a session that searches for vulns and writes up a report, some way for a human to do a review pass on the report, a session that indexes the reviewed findings into tickets, and sessions that fix the fixable issues and submit patches to your repo. The next search session should first read all the open issues so it doesn't duplicate work of earlier sessions.
LESS IS MORE - avoid fancy agent tooling and skills, don't cargo cult from others, build your own tools as you find your own needs. If something can be automated, use the agent to write tools and tests for it, don't just keep prodding the agent to do it.
Can you share with the class the web apps you’ve built this way?