VALOR: Validation for Aerospace LLM Output and Reasoning

Large Language Models (LLMs) offer new opportunities for automating and supporting a wide range of applications. However, despite rapid progress, current LLMs remain prone to factual errors, unstable reasoning, and inconsistent use of context --- limitations that are especially consequential in safety-critical domains. These challenges highlight the need for independent, domain-aware validation frameworks rather than reliance on generic benchmarks or vendor-provided performance claims.

This paper introduces the VALOR framework, a modular evaluation system designed to characterize LLM behavior in question-answering tasks. VALOR provides metrics across four complementary dimensions: accuracy, retrieval quality, robustness, and uncertainty. These enable a more complete assessment of model behavior than traditional closed-ended evaluations.

As a proof-of-concept, we demonstrate the framework using a set of open-ended technical questions based on FAA handbooks. Although developed for aviation, VALOR illustrates a general methodology for independent validation of LLM systems in any safety-critical setting where the reliability and traceability of model outputs are essential. This is useful not only in evaluation of LLM systems, but in their training and hyper-parameter configuration.