Presented at the 4th International Conference on Human Side of Service Engineering, taking place at the 7th International Conference on Applied Human Factors and Ergonomics (AHFE 2016),  Orlando, FL (July 2016)

Commonly used techniques for measuring cognitive workload during human-computer interaction are cumbersome or intrusive to task performance. Physiological (e.g., electrocardiography (Veltman and Gaillard, 1996) and pupillometry (pupil dilation, blink frequency, blink duration (Pomplun and Sunkara, 2003; Tsai, Viirre, Strychacz et al., 2007), saccades (Ahlstrom and Friedman-Berg, 2006)), and neurophysiological (e.g., electroencephalography; (Hirshfield, Chauncey, Gulotta et al., 2009), and functional near-infrared spectroscopy; (Ayaz, Shewokis, Bunce et al., 2012) techniques require on-body sensors, which can be uncomfortable and invasive as well as costly and proprietary. Subjective measures (e.g., NASA task load index (Hart and Staveland, 1988), quantitative workload scale (Spector and Jex, 1998)), although stand-off, can be disruptive when delivered during task performance (Noyes and Bruneau, 2007), or subject to recall bias when delivered following task completion (Yeh and Wickens, 1988). Finally, performance measures are effective for assessing workload directly (Gawron, 2008), however they are most useful for tasks that can be easily categorized as correct or incorrect. Fortunately, technologies for recording user input are becoming increasingly accessible through standard computer input devices and readily available software (e.g., KeyDemon dongles (Amant, Horton, and Ritter, 2007), AppMonitor (Alexander, Cockburn, and Lobb, 2008), Morae, and Camtasia (Morgan et al., 2013)).

In the current work, we are examining the utility of heuristic behavior analysis, including keystroke dynamics, mouse tracking, and body positioning (Mota and Picard, 2003) for measuring cognitive workload during direct interactions between humans and computers. Cognitive workload can be detected with similar accuracy to affective measures using keystroke logging (Vizer, Zhou, and Sears, 2009). Stress can be detected through analysis of pressure applied to a computer mouse (Qi, Reynolds, and Picard, 2001). Finally, dynamic postural changes during cognitively demanding tasks are associated with affective state and engagement level (Mota and Picard, 2003). Increased pressure on the seat is associated with increased engagement, while increased pressure on the back of the chair is associated with boredom (D’Mello, Picard, and Graesser, 2007), and movement decreases with increasing task difficulty (Frank, 2007).

In this study, we are collecting behavioral data during performance of well-validated cognitive tasks to determine the utility of these completely non-intrusive methods for estimating cognitive workload in real-time. We will model these behavioral measures with a range of probabilistic, statistical, and machine learning algorithms demonstrated to be useful for interpreting physiological and neurophysiological signals for real-time estimation of human states. For example, Bayesian Networks (Pearl and Russell, 1998) and statistical averaging are useful for estimating cognitive workload (Bracken, Romero, Guarino et al., 2013; Bracken, Guarino, Dorin et al., 2014). In addition, our custom LearnerBuilder tool (Metzger, Howard, Kellogg et al., 2015), that uses naïve Bayes (boosted), random forest, K-nearest neighbors (boosted), neural network, and support vector machine learning techniques, and our Sum Product Networks approach to Deep Learning are useful for interpreting task difficulty (Bracken, Palmon, Kellogg et al., 2015). This study will enable us to assess the utility of unobtrusive methods for estimating cognitive workload and their applicability to real-world scenarios.