Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly.
If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here.
CLI docs: https://learn.chatgpt.com/docs/security/cli
EDIT: If you'd like to help make this better, we're hiring: https://openai.com/careers/full-stack-software-engineer-cybe...
> Have experience shipping production full-stack products across modern web frontends and backend services.
I'm amazed that the requirements are so low (or at least this vague) for jobs at companies like these.
Has anyone else had the experience of going to an interview and feeling like you were never asked any qualifying questions?
All the questions were easy, your answers were straightforward, you "got them right", but then were not chosen?
I find on the other side, they're also left with dozens of people who "passed" and then it comes down to a pretty arbitrary decision on who gets hired (if we are talking external, no referral, etc.)
I wonder if they can make job descriptions highly specific to filter the shortlist faster and more effectively (to actually get a shortlist).
Anyway end rant. Cool job, hope you fill it.
Not having clear, objective criteria enables arbitrary decisions (not against OP, I mean in general.. and in general I dislike this pattern a lot). On the extreme other end of the spectrum would be 100% objective criteria, and companies being forced to pick a random applicant that matches them. If they want only the best, they have to have high expectations, but be able to actually define them. You say "cultural fit", I say "corruption", let's blow this whole joint.
> You say "cultural fit", I say "corruption", let's blow this whole joint.
It's not necessarily corruption. If there's no conflict between principal and agent, it's fine.
Just like when I sent my butler to go and buy a bottle of wine, he can make arbitrary choices, but that doesn't mean he's corrupt or going against my wishes. I trust his judgement, and since it's a repeated game our incentives are aligned.
I agree, I recently interviewed at Synthiolabs, and they asked me two questions: one about RAG and two about graph RAG. They rejected me even though I answered correctly, and the interviewer was also a college kid.
This sounds like it’s just an app development role, not a security analysis position, so I’m not sure what your complaint is.
Professional app development requires an understanding of security.
I think the evidence contradicts you at this point
If you didn't already know, jobs at highly competitive companies tend to have vague job requirements because they expect to be able to apply your raw intelligence to changing demands quickly. There's no point being hyper-specific about the exact software packages because that's not what they want. What they want is someone who, after talking to an interviewer for 30 minutes, leaves them with the thought "Wow, this person can do anything we need of them. They can probably tell us what we need too and take ownership of large projects. Hire!"
Agree. Being smart, competent, and high in conscientiousness is more important than any highly specific “qualification”. It’s not about checking a bunch of boxes. If you have a track record of getting shit done, you’ll have something to contribute.
This. Being too specific on the stack requirements is a red flag imo.
This looks great, thanks for open-sourcing it!
How does it deal with the current guardrails 5.6 Sol has on finding vulnerabilities? When I use it in the Codex app it would sometimes say it found a vulnerability, but it cannot tell me what it is.
Thanks! You've run into a real limitation: the CLI doesn't bypass the model's cybersecurity guardrails. If GPT-5.6 Sol finds a vulnerability but refuses to explain it, switching from the Codex app to the CLI won't automatically fix that.
For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals depending on the model and the account or organization where access is provisioned. It isn't a blanket bypass.
If you're an open-source maintainer, you can apply for conditional Codex Security access here:
https://openai.com/form/codex-for-oss/
For enterprise teams, the public Daybreak onboarding guide is here:
https://help.openai.com/en/articles/20001261-enterprise-dayb...
If you have an example of "found a vulnerability but won't tell me what it is," I'd love to take a look too. You can send it to use with /feedback (or message me).
>> For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals
Or perhaps a better option is to use something like Kimi K3 and cancel the GPT subscription altogether.
I tried it, it started a scan but stopped after hitting the rate-limit of my account. It gave up after just a minute of retrying (rate limits are tokens per minute, so... :P).
It said "Partial output was kept at <...>", but I dont see a obvious way of picking it up in a new scan? (The failed run cost me ~$13)
Yeah, you're right. A per-minute rate limit shouldn't kill a scan after a minute, and "partial output was kept" makes it sound like you can pick up where you left off. You can't yet, unfortunately. --max-cost can limit estimated spend, but we still need proper retries and resume. Sorry you spent $13 finding that out. Please send me an email and I'll help make it right.
When would I use this over the plugin in codex? Which I think can be invoked from cli as well
The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration.
We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build.
Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.
Does it require hitting OpenAI's APIs or can one also stand up a local OpenAI compatible LLM endpoint?
We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.
Exciting! Is there any open GitHub issue we can track?
Not yet, unfortunately. We only just opened the repo, and there isn't a public issue specifically tracking local or OpenAI-compatible endpoint support.
The issue tracker is here: https://github.com/openai/codex-security/issues
If you open an issue with the endpoint or model you want to use, I'd be happy to follow up there.
Thanks! https://github.com/openai/codex-security/issues/52
Been watching your progress for a while, glad OpenAI have looked after you and the team and you still get to ship!
Thank you, that means a lot. Being able to keep building practical, open-source security tooling was important to us.
Really glad we got to ship this, and there's still a lot we want to improve in Codex Security and in Promptfoo!
How does it fare against its own codebase?
> co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.
> Thanks for checking this out and for flagging the auth issues.
Offtopic, but this right here is why I don't believe any marketing around "great amazing models that one-shot everything and programmers are no longer needed".
You just have to look at what these labs routinely produce, and their own products.
Edit to respond to @simonw whose comment I saw before he retracted it ;)
This comment is tied directly to consistent continuous claims by the LLM labs. Their own products disprove their own claims, and it would indeed be nice if fewer people believed them :)
Why does this need an entirely separate repo instead of being a feature in the existing Codex project?