Terry Patten1, Rachel Amey2, Joanne Barnieu3, Clarence Dillon3, Jennifer Harvey3, Sean Shiverick3, Michael Smith3, Steve Hookway1
Paper Session, MODSIM 2023, Norfolk, VA (May 2023).
1 Charles River Analytics
2 U.S. Army Research Institute for the Behavioral and Social Sciences
3 ICF
Rapid and reliable individual-level assessments are critical to developing effective and capable Army Soldiers and meeting the needs of modern warfighters. Changes in required skillsets or equipment require more frequent updates to training and the creation of new training courses, which, in turn, leads to creation of new assessments to measure knowledge, skills, and abilities (KSAs). Traditional methods of assessment development involving manual item construction are labor-intensive, time-consuming and often costly. The Army is investigating methods to scan available training text (e.g., field manuals, lesson plans) to automatically generate assessments, reducing the need for human involvement. While syntactic and neural network methods for automated assessment are currently prevalent, the team is applying novel semantic information-extraction technology to address the limitations of these approaches. Using the frame-based approach pioneered by Berkeley, it is possible to generate a variety of grammatically and semantically sound stems and response options unconstrained by the original wording in the instructional material. Additionally, this approach does not require large amounts of language-model training data, is capable of assessing higher levels of reasoning, and allows model parameters to be adjusted directly. The research team has proven the feasibility of this automated approach and developed a system prototype which generates items from test-bed training source material concerning a type of Army equipment.
For More Information
To learn more or request a copy of a paper (if available), contact Terry Patten.
(Please include your name, address, organization, and the paper reference. Requests without this information will not be honored.)
