Ridge Security Publishes First-of-Its-Kind Benchmark Comparing Leading AI Models for Autonomous Red Teaming
Eight-model study finds the "smartest" LLM isn't automatically the best autonomous penetration tester, highlighting agent architecture, execution and verification as critical to AI-powered offensive security
SILICON VALLEY, Calif., September 03, 2026--Ridge Security, a leader in AI-powered offensive security and continuous security validation, today published the industry's first public benchmark evaluating multiple leading large language models (LLMs) in autonomous penetration-testing workflows. The results show the model is only half the equation. How an AI harness reasons, executes, adapts, and validates its actions matters just as much, especially for enterprise-grade requirements.
Using its RidgeGen™ offensive-security harness, Ridge Security evaluated eight AI models across 96 model-target test runs against intentionally vulnerable environments, including VAmPI, Metasploitable3, OWASP Juice Shop and OWASP WebGoat. The benchmark tracked whether each model could carry that through the full workflow of reconnaissance, hypothesis testing, payload adaptation, exploitation and verification without stalling out partway through.
The results showed significant differences in security coverage, cost and efficiency:
Grok 4.5 led on cumulative coverage at 77%
Claude Opus 4.6 reached 63% coverage, at an estimated $217 per run
Gemini 3 Flash achieved 52% coverage at approximately $5.42 per run
GPT-OSS-120B delivered the benchmark's highest peak efficiency at 16.9 findings per million tokens, at approximately $2.32 per run
The findings challenge the assumption that the highest-performing general-purpose LLM will automatically deliver the strongest autonomous security agent.
"Security teams are beginning to ask which LLM is smartest, as though that answer determines which AI system will be the best penetration tester," said Lydia Zhang, President of Ridge Security. "Our research shows that autonomous offensive security is a systems problem. The model needs an architecture around it that can manage execution, adapt to what it discovers and verify that a finding is real."
The Harness Matters
Autonomous penetration testing requires more than cybersecurity knowledge or the ability to generate exploit code. An agent must maintain state across an attack sequence, adapt when a path fails, execute tools, chain weaknesses together and ultimately prove that a vulnerability is exploitable.
Ridge Security's benchmark shows the agent harness is a must-have layer in AI-powered offensive security. RidgeGen is designed to separate reasoning from execution and verification: the model reasons, the harness controls execution, and independent validation confirms whether a finding is real.