ABOARD
A testing framework to improve railroad safety
through human-AI interaction analysis
A testing framework to improve railroad safety through human-AI interaction analysis
Assessment for Better Operator-AI-centered Research and Development (ABOARD)
Recent advances in rail technology include semiautonomous rail operations to improve the safe and efficient transport of passengers and freight. As AI and autonomous systems are being deployed more commonly across the transportation sector, it is essential to consider and test potential issues with operator-AI interaction. As rail operators develop and evaluate their algorithms, they also need to identify the wider implications of integrating AI with human operators in the real world.
The goal of ABOARD is to develop a testing and evaluation (T&E) framework for assessing human-autonomy interactions in intelligent rail software applications. The project will allow a better understanding of human factors to improve safety and enable more thoughtful and safer deployment of autonomous technology by addressing the challenges of integrating artificial intelligence (AI) algorithms and human operators.
“Human-AI teaming is an important component of deploying AI-enabled technology, and you need to have a plan for testing and evaluating it. Properly evaluating a human-in-the-loop will give you information and feedback on how to improve the software in the iterative design and deployment process.”
Mandy Warren
Senior Scientist and Principal Investigator on ABOARD
Building T&E into AI development
The Charles River team is leveraging previous experience in human-machine interface development and cognitive systems engineering methods to design a concept of operations (CONOPS) for a software-enabled simulation test bed that AI developers can use when developing and testing AI-enabled technologies to evaluate the human-machine interaction for operational deployment.
The team is partnering with industry experts, including a locomotive engineer and research leaders in human-AI teaming, to ground the T&E approach in realistic rail-related requirements. Additionally, they are forming a working group to develop best practices and principles for human-AI teaming T&E.
Performance metrics related to human-AI interactions were intentionally built into the ABOARD framework as a core design element. Including these T&E steps in ABOARD will permit software engineers to identify potential issues or concerns with the human-AI teaming early in the software development process.
Defining best practices for human-AI teaming
Building performance metrics directly into ABOARD ensures that testing and evaluation are integrated from the beginning of the technology development cycle. Instead of treating evaluation as an afterthought, the framework emphasizes defining measurable requirements early on to guide the design and development process.
With the capability to evaluate AI teaming software in a simulation environment, ABOARD will provide a framework for engineers to improve human-AI integration and establish rail and transportation industry best practices and requirements that will enable safer deployment of autonomous technologies.
While this project focuses on rail operations, the findings will be helpful for the broader transportation industry as it will allow AI developers to better understand how to safely deploy and monitor AI technology with human-in-the-loop oversight.
Contact us to learn more about ABOARD and our other human factors and AI capabilities.
This material is based upon work supported by the Federal Railroad Administration (FRA) under Contract No. 693JJ6-24-C-000019. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the FRA.
