A Survey Protocol to Assess Meaningfulness and Usefulness of Automated Topic Finding in the NASA Aviation Safety Reporting System

Context: The NASA Aviation Safety Reporting System (ASRS) is a voluntary confidential
aviation safety reporting system. The ASRS receives reports from pilots, air traffic controllers,
flight attendants and other involved in aviation operations. The reports are de-identified and
coded by ASRS expert safety analysts and a short descriptive synopsis is written to describe
the safety issue. The de-identified reports are then disseminated to the aviation community in
a number of ways including entry into an online database, Safety Alert Bulletins and For Your
Information Notices, and the CALLBACK newsletter. Key to these publications are the timely
processing (de-identification, coding and summarization) of new reports, which is currently
done by ASRS expert safety analysts. Thus, we believe topic modelling could decrease effort in
ASRS, if topics are comprehensible.

Aim: We propose a methodology to evaluate whether automated topic finding using topic
modelling provides meaningful and useful topics.
Method: We extend the total error survey methodology to evaluate user topic comprehension of machine learning outputs. To accomplish this we performed a literature review to identify existing methods and define a construct for topic comprehension, utilizing existing
ASRS synopsis writing practices to more precisely define meaningfulness and usefulness.

Results: A survey protocol was created that addresses the limitations of other survey
protocols found in the literature review, which we found lacking in rationale and clear protocol
definition.

Conclusion: The surveying of user understanding in machine learning outputs presents
challenges due to the explosion of parameters to control for and the lack of systematic approach
presented in the literature. More reproducible work and survey protocols are needed in the
literature and our work is one step towards that direction.