Aviation-Specific Large Language Model Fine-Tuning and LLM-as-a-Judge Evaluation

This paper presents a scalable approach to improve domain-specific understanding by developing an aviation-focused large language model (LLM). While recent LLM advancements, like supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF),have improved general conversational abilities, these methods are costly and difficult to scale in aviation due to limited labeled data. To address these challenges, we applied a self-supervised fine-tuning approach to provide responses to aviation-related query generation without requiring human-labeled question-answer pairs. During fine-tuning, we incorporated the LLM-as-a-judge framework to automatically identify the optimal training parameter combinations. Additionally, to evaluate model performance, we leveraged the same LLM-as-a-judge framework by utilizing a larger LLM to assess the responses generated by both the base and fine-tuned models on aviation-specific questions. This methodology provides a scalable, automated, and high-quality solution for domain-specific language modeling in aviation.