The Department of Transportation awarded Charles River Analytics a contract to advance the Federal Railroad Administration’s research initiatives in support of rail safety. The goal is to develop a testing and evaluation (T&E) framework for assessing human-autonomy interactions in intelligent rail software applications. The project will allow a better understanding of human factors to improve safety and enable more thoughtful and safer deployment of autonomous technology by addressing the challenges of integrating artificial intelligence (AI) algorithms and human operators.
Recent advances in rail technology include semiautonomous rail operations to improve the safe and efficient transport of passengers and freight. As AI and autonomous systems are being deployed more commonly across the transportation sector, it is essential to consider and test potential issues with operator-AI interaction. As rail operators develop and evaluate their algorithms, they also need to identify the wider implications of integrating AI with human operators in the real world.
“Human-AI teaming is an important component of deploying AI-enabled technology, and you need to have a plan for testing and evaluating it,” said Mandy Warren, Senior Scientist at Charles River Analytics and Principal Investigator on the project. “Properly evaluating a human in the loop will give you information and feedback on how to improve the software in the iterative design and deployment process.”
Warren and her colleagues are designing and developing an Assessment for Better Operator-AI-centered Research and Development (ABOARD). They are leveraging previous experience in human-machine interface development and cognitive systems engineering methods to design a concept of operations (CONOPS) for a software-enabled simulation test bed that AI developers can use when developing and testing AI-enabled technologies to evaluate the human-machine interaction for operational deployment.
The Charles River team is partnering with industry experts, including a locomotive engineer and research leaders in human-AI teaming, to ground the T&E approach in realistic rail-related requirements. Additionally, they are forming a working group to develop best practices and principles for human-AI teaming T&E.
Warren and her team intentionally built performance metrics related to human-AI interactions into the ABOARD framework. Including these T&E steps in ABOARD will permit software engineers to identify potential issues or concerns with the human-AI teaming early in the software development process.
“By building these performance metrics directly into ABOARD, we’re not leaving testing and evaluation to the end of the technology development cycle,” Warren said. “Right from the outset, as you’re developing your requirement set for your technology, you should be deriving performance metrics that you can include in your testing and evaluation program.”
With the capability to evaluate AI teaming software in a simulation environment, ABOARD will provide a framework for engineers to improve human-AI integration and establish rail and transportation industry best practices and requirements that will enable safer deployment of autonomous technologies.
While this project focuses on rail operations, the findings will be helpful for the broader transportation industry as it will allow AI developers to better understand how to safely deploy and monitor AI technology with human-in-the-loop oversight.
Contact us to learn more about ABOARD and our other human factors and AI capabilities.
This material is based upon work supported by the Federal Railroad Administration (FRA) under Contract No. 693JJ6-24-C-000019. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the FRA.
