Measuring the Operational Utility of HSCB modeling and Methods

Pfautz, J., Carlson, E., Prue, B., and Koelle, D.

Presented at the 4th International Conference on Applied Human Factors and Ergonomics (AHFE) San Francisco, CA (July 2012)

The HSCB modeling community has shown a increased commitment to ensuring that computational representations of human socio-cultural phenomenon can be scientifically validated. Yet the challenges of performing rigorous validation studies have been well documented, forcing a re-examination of what it means for a model to be “valid.”  The term application validity provides a utilitarian perspective; that is, a model needs only be valid in the context of a specific application. Other researchers have espoused the operationalization of not only of HSCB models, but also HSCB knowledge and methodsranging from simply helping operators to understand potentially relevant variables, to providing guidance on known causal relations between variables, to enabling operators to create, share, and validate their own model.

Assuming willingness to adopt such utilitarian perspectives in the face of the slow but steady pace of fundamental research in HSCB modeling and simulation, a number a key question must be considered:  How do we understand and measure human performance when aided by some (presumably at least quasi-computational) HCSB system?  Fortunately, the human factors community has been focused on the scientific pursuit of this very question for decades. Current research focuses on a holistic view of the “socio-technical system” – the human operator, the system or systems they use, the socio-organization context, as well as other salient characteristics of the work environment.  The principles and practices of cognitive systems engineering, when applied, provide the basis for analyzing the components of human-HSCB system activities in terms of not only the broad application of the system, but also in terms of the specific tasks performed internally and across humans and the system (e.g., cognitive tasks, coordination tasks, automated tasks). Clearly, if a utilitarian perspective on HSCB is to be adopted, such methods are needed to help systematically decompose what is “useful.” This approach has the added benefit of supporting the measurement of human-system performance, since individual and combined tasks are decomposed to a level that makes identification of metrics relatively straightforward (e.g., “Did Raoul make better decisions?” vs. “Did Raoul properly incorporate the causal relationship between tribal affiliation and propensity for charitable giving as he decided to seek additional information about financial transactions supporting an adversary’s campaign emphasizing corruption?”)

In our research and development efforts, our focus on providing operationally relevant systems forced us to adopt a utilitarian perspective, and therefore led us to identify key areas for deeper fundamental research into how HSCB models and methods can impact human-system performance. For example, understanding just when and how an operator may choose to incorporate HSCB model results into their reasoning is a highly complex process, involving the consideration of meta-information about those results and how they were produced (e.g., What data was used? Who created this model? What assumptions are inherent in the model? How long will it take to understand the model results?). Similarly, identifying the space of relevant HSCB methods requires careful consideration not only of the tasks, but also any number of different HSCB capabilities and technologies, beyond the more typical focus on models.  Also, the communication of HSCB data, given its propensity for being considered “soft,” vague, and/or ambiguous (particularly in military contexts) is fraught with unexplored complexities.

In this paper, we describe these areas for further research as well as current progress, based on our experiences in operationalizing HSCB models and methods (as well as other related research) for communities focused on non-traditional military operations focused on shaping human behavior.

For More Information

To learn more or request a copy of a paper (if available), contact J. Pfautz.

(Please include your name, address, organization, and the paper reference. Requests without this information will not be honored.)