How to Tell If a Vulnerability Report Is AI-Generated

Five checks that take under a minute each, before you spend an hour reading the wrong thing carefully.

curl shut down its bug bounty program in early 2026 after AI-generated reports pushed submission volume to roughly 8x normal — about 20% of them citing vulnerabilities that simply don't exist in the code. HackerOne now puts the invalid-submission rate across its whole platform at 60-80%. Bugcrowd's triage queues have grown over 300% in weeks, almost entirely low-quality AI submissions. If you maintain anything with a security contact, this is already your problem.

The good news: most AI-generated reports share the same failure mode, and it's checkable without reading the whole thing.

1. The file path doesn't exist

A model writing a plausible-sounding report will often cite a file path that sounds right for the project's structure but isn't actually there — a renamed file, a path from a different version, or one invented wholesale. Open the repo and check the path exists before reading further.

2. The line number is past the end of the file

Reports that cite "line 847" in a file that's 300 lines long are an instant tell. This happens constantly because the model isn't looking at the real file — it's generating a plausible-looking citation.

3. The function name doesn't appear anywhere in the file

Same failure mode as the line number: a named function that isn't defined, isn't called, isn't referenced anywhere in the file the report points to.

4. The report is confident but has no concrete pointers at all

The inverse tell: a report that describes a vulnerability in vivid, technical-sounding language but never names a specific file, line, or function is unfalsifiable by design — there's nothing to check, which is itself worth noticing. That doesn't make it fake, but it means you can't clear it in a minute; it needs a human to actually read the code.

5. The severity language doesn't match the actual claim

"Critical remote code execution" attached to what turns out to be a theoretical issue requiring local access and three other preconditions is a pattern specific to reports optimized to look scary rather than to be accurate.

Checks 1-3 can be automated — that's the whole idea

File existence, line-count bounds, and function-name presence are all mechanical checks against your repo's actual current state. No LLM judgment call needed, no vibes — a file either exists at that path or it doesn't. That's what TriageShield does: paste a report and a GitHub repo URL, and it runs exactly these checks against the real repo and shows you which claims don't hold up, in seconds instead of an hour.