Sean Guarino1, David Kelle1, Curt Wu1, K. Raghav Bhat2, Robert Gutzwiller2, Max Slocum3, Michael Sieffert3, Michelle Neisser4
Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando, Florida (3 December 2025)
Gaining a better understanding of the cognitive constraints of cyberattackers can drive the development of effective proactive cyber defenses. Studying hacker behavior, however, requires the development of highly realistic and well-instrumented cyber testbeds. Here, we describe key simulation challenges that we experienced while developing testbeds to execute five human subjects studies analyzing the ability to exploit cognitive biases of cyber adversaries, and the solutions we developed to address these problems. Specific challenges addressed include the unique experimental requirements for testbed configuration and management across numerous human subjects, requirements for generating content and variation that support realism and avoid training effects in within-subject studies, requirements to ensure participants can complete studies in the time constraints provided, and requirements to effectively instrument the testbed to better understand hacker behaviors. For example, one area of challenge that we confronted was supporting a trade-off between the need for an open-world, realistic cyberattack context, and the need to guide attackers to complete specific tasks so that we could collect pertinent behavior and performance data. Addressing these challenges allowed us to streamline our development process, to more rapidly construct testbeds for our later and future studies. Specifically, we describe the following: (1) how we configured sets of testbeds to ensure experimental consistency across subjects as they executed each trial twice; (2) how we used large language models (LLMs) to generate files and other content within each testbed, populating the testbeds with realistic, differentiated artifacts rather than using existing, recognizable datasets; (3) how we used a variety of methods to manage the time and progress of attackers across a range of skill levels; and (4) our development of a data collection pipeline to enable the analysis of both behavioral indicators of susceptibility and outcome-based performance metrics.
1 Charles River Analytics Inc.
2 Arizona State University
3 Assured Information Security
4 SimSpace Corporation
For More Information
To learn more or request a copy of the paper, contact Sean Guarino.
(Please include your name, address, organization, and the paper reference. Requests without this information will not be honored.)