AI-Generated API Test Suites vs Real-World API Vulnerabilities: An Empirical Study

Abstract

Application Programming Interfaces (APIs) form the backbone of modern distributed systems, yet they remain prime targets for security exploits. While AIpowered test generation tools promise automated, intelligent coverage of API endpoints, their effectiveness against real-world vulnerability classes remains underexplored. This paper presents an empirical study evaluating the detection capabilities of AI-generated API test suites against a curated dataset of real-world API vulnerabilities, including OWASP API Security Top 10 issues such as Broken Object Level Authorization (BOLA), injection flaws, and authentication bypasses. We benchmark multiple AI-driven testing frameworks— including LLM-based payload generators and specification-aware test synthesizers—against vulnerability-oriented inspection tools like VOAPI2 and traditional fuzzers. Our experiments reveal that while AIgenerated tests achieve 60–85% path coverage and reduce manual effort by up to 80%, they systematically miss 30–50% of high-severity, context-dependent vulnerabilities that require stateful request sequences or business-logic exploitation. We identify key failure modes: over-reliance on schema conformance, inability to model multi-step attack chains, and blind spots in authorization boundary testing. Based on these findings, we propose a hybrid testing paradigm that combines AIgenerated breadth with targeted, vulnerability-patterndriven depth, and we release an open benchmark suite of 50+ real-world vulnerable API endpoints for future research. Our results underscore that AI-generated test suites are powerful accelerators but insufficient as standalone security assurance mechanisms for production APIs.