Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing

This paper presents a comprehensive evaluation of AI agents against human cybersecurity professionals in live enterprise penetration testing. It highlights the capabilities of AI in discovering vulnerabilities and enhancing cybersecurity defenses.

The cybersecurity industry faces a growing disparity between the number of sophisticated threats and the available workforce of skilled penetration testers. Manual penetration testing is expensive, slow, and unscalable, leaving many applications undertested. Existing automated scanners (DAST) lack the reasoning capabilities to find complex logical vulnerabilities, creating a need to evaluate if current AI agents can bridge this gap.

The authors established a controlled benchmarking environment containing a suite of web applications with diverse vulnerability types (SQLi, XSS, Logic Flaws). They deployed AI agents configured with ReAct-style prompting and access to standard security tools (web browsers, proxy scanners, terminal access). A control group of human participants, categorized into Junior, Mid-level, and Senior profiles, performed penetration tests on the same targets. Metrics collected included Time-to-Pwn (time to exploit), Cost per Vulnerability (calculating tokens vs. hourly wages), and False Positive Rate. The agent architecture utilized GPT-4o as the reasoning engine.

AI agents demonstrated a cost reduction of up to 20x compared to human testers for discovering low-to-medium complexity vulnerabilities. In terms of speed, agents were able to scan and exploit targets significantly faster than human juniors. However, the success rate for agents dropped sharply on targets requiring multi-step reasoning or novel exploit chains, where Senior human testers maintained a high success rate. Agents also showed a higher tendency for 'rabbit holes' (pursuing dead ends) without human intervention. The study quantified that while agents can replace about 50-60% of routine testing tasks, they cannot yet replace the critical thinking of a mid-to-senior level professional.

The paper provides a much-needed empirical grounding for the hype surrounding AI in cybersecurity. By directly comparing agents to humans of different skill levels, it offers a nuanced view that avoids binary 'AI is useless' or 'AI replaces everyone' conclusions. However, the definition of 'real-world' is necessarily constrained to the test environments created, which may lack the chaotic complexity of legacy enterprise networks. The study is technically sound but could benefit from a longer longitudinal study to assess agent adaptability over time.