Building the most realistic bug bounty CTF ever run at DEFCON

The Bug Bounty Village CTF is an official DEFCON contest, one of the recognized competitions at the world's most influential hacker conference. The Village's mission is to bridge the gap between security researchers and vulnerability disclosure platforms through hands-on technical challenges. They needed a CTF that felt like real bug bounty, not a typical CTF.

Bug Bounty Village co-founders Harley Kimball and Ariel Garcia trusted us with a concept that had never been done before: a bug bounty CTF. We had to invent the game logic from scratch, balancing the realism of bug bounty with the gamification of a CTF. We met with triagers from HackerOne to understand their workflow. We studied real genomics companies. And we built something that made experienced bug bounty hunters say it felt like the real thing.

GeneQuest is a fictional direct-to-consumer genomics company inspired by 23andMe that sells DNA kits and delivers ancestry, trait, and health-risk insights. We built it from the ground up as a fully functional web application with a complete brand identity: marketing copy, product pages, user dashboards, and a research portal. Every detail was designed to make players forget they were in a CTF.

Under the surface, GeneQuest spans 10 microservices with 80+ endpoints, built across 6 frameworks and 4 programming languages. The 26+ vulnerabilities range from simple IDORs and auth bypasses to multi-step attack chains requiring correlation between distant sources and sinks. Every vulnerability was modeled after real bug bounty reports.

GeneQuest was built LLM-native. Two AI systems were embedded directly into the application, not bolted on as separate challenges, but integrated the way modern web apps actually use AI.

We challenged Ethiack to deploy their autonomous security agent against GeneQuest to see how an AI would perform against a target designed for humans. The results were striking.

In just 4 hours, the agent found 4 flags, including an unintended local file inclusion vulnerability that no human player discovered across the entire 3-day event. The best human researcher found 14 out of 26 vulnerabilities over 3 days (53%). The agent found its 4 in a fraction of the time.

This experiment directly informed our AI Benchmarking service. GeneQuest proved that purpose-built vulnerable labs are the most effective way to evaluate AI model capabilities in realistic conditions. We now offer this as a dedicated service: building custom vulnerable environments for organizations who need to understand what their AI can and cannot do before deploying it.