DeBERTa-AT: a DeBERTaV3 Variant Fine-Tuned on Air Traffic Data

Large language models (LLMs) offer a powerful platform and can leverage tools to extract relevant information and provide recommendations for air traffic users. These can range from classification of voluntary safety reports to discovering shared successful corrective actions and creating more accurate transcriptions of air traffic control conversations. However, using LLMs requires leveraging the knowledge contained in legacy aviation datasets, which can be time-consuming and compute-intensive. This paper describes the creation of DeBERTa-AT, a variant of the DeBERTaV3 model fine-tuned on air traffic data. DeBERTa-AT continues pretraining using the replaced token detection (RTD) pretraining task with gradient-disentangled embedding sharing (GDES) to offer improved performance and faster training convergence on downstream tasks in the aviation field. DeBERTa-AT is evaluated on two downstream binary classification tasks: aviation-specific constraint classification and document classification, with text datasets created from letters of agreement (LOAs). During downstream fine-tuning, improvements were shown in performance as compared with theDeBERTaV3-base model with frozen embeddings. The results show that models combined with sample-efficient continual pre-training can offer substantially better results on domain specific data when less training data is available.