When attackers upgrade their AI every month, annual pentests can’t keep up
Author: Dr. Karsten Nohl, Chief Innovation Officer, Allurity
Most organisations still test their software the way they did a decade ago: once or twice a year. Usually, an external team probes the application, files a report, and leaves. That model worked when attacker capability changed slowly. It no longer does.
AI has led to a step change in testing capability – not by finding more (not yet, anyway), but by finding issues faster. An AI agent can scan for and exploit weaknesses within minutes and never get tired. AI and our understanding of how to leverage it for testing improve every month.
The question for security leaders is no longer whether adversarial AI will be pointed at your applications. It is who points it first – you or the hacker.
What Allurity research showed
To understand how close this future already is, our research team ran a study putting today’s leading AI models to work on real production codebases: a fintech dashboard securing large portfolios, and an open-source voting platform. We measured how many genuine security issues each model found, how reliable those findings were, and – importantly for regulated industries – whether this could be done without sending proprietary source code to a cloud service. All at low cost with no need for Mythos or any other export-restricted model.
Three findings stood out; all with direct business implications:
Process matters more than raw model power. A well-designed testing method made every model dramatically better — often doubling what it found. Surprisingly, compact models running on a laptop, guided by a good process, produced findings competitive with frontier cloud models, even Claude’s mysterious models. The lesson for businesses is reassuring: world-class AI testing does not require handing your source code to cloud LLM providers. With the right approach, it can run on hardware you control.
Sensitive code can stay in-house. We confirmed a practical setup where the source code never leaves the machine. A cloud model designs the review and assembles the final report using only high-level metadata, while a local model does the actual reading of the code. For finance, government, healthcare, and critical infrastructure — where data residency and confidentiality are non-negotiable — this removes the usual trade-off between strong AI testing and keeping proprietary code private.
No single model finds everything. Each AI model we tested caught a different slice of vulnerabilities. One excelled at architectural reasoning, another at tracing how data moves through an application, others at spotting line-level coding mistakes. The combined set of findings was far larger than any one model produced alone. In practice, running a “second opinion” model genuinely expands coverage — which means any testing that relies on a single tool, or a single point-in-time, is leaving unnecessary risk on the table.
What this means for your testing program
None of this replaces the pentest you already run. Your annual or bi-annual engagement still has its place: it satisfies regulators, it catches some bug classes that AIs still miss, and it allows the testers to re-confirm the power (and limitations!) of AI-based testing. The problem is simply that one snapshot a year cannot keep pace with a threat that shifts with every model release.
This is the gap Allurity’s AI Hacking Readiness service is built to close. It is a managed, continuously updated assessment layer that puts current hacker tools on your side of the fence — month after month, model after model. We test your applications with the same AI capabilities adversaries use, re-test within two business days when a significant new model lands, and have our pentest experts verify critical findings before they reach your team. You receive a living view of where adversarial AI focuses on your application, a tracked feed of findings that goes straight into your existing workflows, and — for sensitive code — the option to run everything locally.
The result is a security program that stays current as fast as the threat does, showing you monthly how much security buffer remains until AI models know everything about your applications that a skilled pen tester does. For the moment, that buffer is still there.
Curious where adversarial AI would focus on your applications? We’ll run one of them through the full pipeline so you can see the threat model, the findings, and the delivery format before committing to anything. Get in touch.
Read the full technical study, Beyond Fable, on the SRLabs research blog.