Nicholas Carlini - Black-hat LLMs | [un]prompted 2026

This matches with what Nicholas Carlini from Anthropic said a the [un]prompted conference - ▶ this video . Very worth watching.

This is the talk by Carlini, only half-way through it but matches what you described i.e. run the prompt on each file: ▶ this video

About five minutes in in this video: ▶ this video They also say publicly in their Opus 4.6 post ( link ): >In this work, we put Claude inside a “virtual machine” (literally, a simulated computer) with access to the latest versions of open source projects. We gave it standard utilities (e.g., the standard coreutils or Python) and vulnerability analysis tools (e.g., debuggers or fuzzers), but we didn’t provide any special instructions on how to use these tools, nor did we provide a custom harness that would have given it specialized knowledge about how to better find vulnerabilities. This means we were directly testing Claude’s “out-of-the-box” capabilities, relying solely on the fact that modern large language models are generally-capable agents that can already reason about how to best make use of the tools available.

The technique Anthropic uses was demonstrated by Nicholas Carlini in a talk he gave 2 weeks ago and it's very simple, when asking LLMs to review code, ask them to review one file in a single session. Here is the video with the timestamp (watch through to ~5:30, they show two different ways of prompting claude). ▶ this video ▶ this video IMO the big "innovation" being shown by Mythos is the effectiveness with prompting LLMs to look for security vulnerabilities in specific files one at a time and automating this prompting with a simple script. Prompting Mythos with a single file per session is why I suspect it cost Anthropic $20k to find some of the bugs in these codebases. I know this same technique is effective with Opus 4.6 and GPT 5.4 because I've been using it on my own code. If you just ask the agent to review your pr with a low effort prompt they are not exhaustive, they will not actually read each changed file and look at how it interacts with the system as a whole. If the entire session is to review the changes for a single file, the llm will do much more work reviewing it.

I would suggest watching Nicholas Carlini's talk and Heather Adkins and Four Flynn's talks from unprompted: ▶ this video link My takeaway is that fuzzing is not just complementary, it also gives a stronger AI a starting point. But AI is generally faster and better.

I would've basically agreed with you until I'd seen this talk: ▶ this video Maybe a bad example since Nicholas works at Anthropic, but they're very accomplished and I doubt they're being misleading or even overly grandiose here See the slide 13 minutes in, which makes it look to be quite a sudden change

You should watch this talk by Nicholas Carlini (security researcher at Anthropic). Everything in the talk was done with Opus 4.6: ▶ this video

I'm sure they do, yet the models really are getting scarily good at this. This talk changed my view on where we're actually at: ▶ this video

I wonder about the "video running in the background" during qna of the talk: ▶ this video Did he write an exploit for the NFS bug that runs via network over USB? Seems to be plugging in a SoC over USB...?

▶ this video is the presentation itself. The prompts are trivial; the bug (and others) looks real and well-explained - I'm still skeptical but this looks a lot more real/useful than anything a year ago even suggested was possible...

What I found is that the "For You" algorithm on X was better than I thought. Yes, it starts off as terrible, and remains terrible without very careful steering. I'm treating X as read-only and not interacting. But after liking a few posts by Amanda Askell, Simon Willison, Ethan Mollick, Boris Cherny and others, I was immediately getting far better AI information than I was on Bluesky, from accounts beyond the ones where I'd liked their posts. That isn't necessarily a high bar, but at least it was people actively using AI, people who are positive about the things that can be made with it. I'm more likely to hear about new model releases that I won't hear about elsewhere. At least on X, I'm going to hear people talking about Claude Mythos, and I'll hear about the Nicholas Carlini talk [1] where Claude Mythos is finding & reporting CVEs in the Linux kernel. On Bluesky, my time was being spent fighting off anti-AI university professors who were still in the "stochastic parrot" phase. They were genuinely convinced AI models couldn't even write code that compiles, let alone entire working programs that solve problems. I'm not trying to persuade you or anyone else to use X, and I'd really rather not be using X myself. [1] ▶ this video

Intrinsically simple and straight forward problems are easier to secure even with mediocre or bad code. They've already shown that Opus 4.6 can find and report on very sophisticated security issues[0] so I'm not sure that analysis (and perhaps especially security analysis) is the biggest issue with LLMs. Mind you, I'm not using LLMs for professional programming since I prefer knowing everything inside and out in the code that I work on, but I have tried a bunch of different modes of use (spec-driven + entire implementation by Opus 4.6, latest Codex and Composer 2, and entirely "vibecoded", as well as minor changes) and can say that for trivial in-house things it's actually usable. Do I prefer to rewrite it entirely manually if I want something that I actually like? Yes. Do I think that not everything needs to be treated that way if you just want an initial version you can tinker with? Also yes. 0: ▶ this video

This very question was asked to Nicholas Carlini from Anthropic at this talk: ▶ this video The answer is complex, worth watching the video. But mainly, they don't know where to place the line. Defenders need tools, as good as attackers. Attackers will jailbreak models, defender might not, it's the safeguard positive in that case? Carlini actively asks the audience and community for "help" in determining how to proceed basically.

Obviously no guarantees that it's exactly what was done in this case, but he talked about his general process recently at a conference and more in depth in a podcast: ▶ this video link It pretty much is just "Claude find me an exploitable 0-day" in a loop.

The talk "Black-Hat LLMs" just came out a few days ago: ▶ this video Looks like LLMs are getting good at finding and exploiting these.

It's this talk right here: ▶ this video 7 minutes in, he shows the SQLI he found in Ghost (the first sev:hi in the history of the project). If I'd remembered better, I would have mentioned in the post: * it's a blind SQL injection * Claude Code wrote an exploit for it. Not a POC. An exploit.