Characterization Study of TestBed Infrastructure Performance in a Distributed Simulation Environment: Baseline Analysis

Characterization of the performance of Air Traffic Management Exploration (ATM-X) TestBed integration environment has been investigated and documented for one system configuration for progressively increasing traffic. Several statistical parameters were used to assess the performance of the TestBed distributed system such as mean, standard deviation, skewness, and kurtosis of latency, and update rate for aircraft state messages that are transmitted through the simulated system under investigation.

It is necessary to assess the performance characteristics of distributed systems in terms of the indicated statistical parameters mentioned above. It is critical to verify the system performance with respect to a researcher’s required system performance. Computer host specifications are documented in terms of Central Processing Unit (CPU) clock speed and core count. Transmission Control Protocol/ Internet Protocol (TCP/IP) message protocol was used for data transmission. The system network topology also contributes to the latency and update rate variations from the one imposed by the data source.

The motivation for selecting the TestBed infrastructure as the focus of this study can be attributed to the number of services and capabilities it provides that help simplify the process of preparing and conducting a simulation. These capabilities include an easy to use GUI for simulation configuration, access to TestBed library by the end-user of other simulation software components, a modular adapter paradigm that allows simple connectivity of external software to TestBed, connectivity with other simulation laboratories, and a Software Development Kit (SDK) for quicker development.

Two types of traffic generators, Air Traffic Generator (ATG) and Multi Aircraft Control System (MACS) were used to generate messages that were injected into the TestBed distributed environment. Eight different air traffic scenarios with progressively increasing loads were generated for each air traffic simulator. The corresponding air traffic loads between the two simulators had an identical number of aircraft per scenario, but different flight plans.
It was observed that the performance of MACS degraded for air traffic scenarios containing more than 200 aircraft (37.5 KB/s nominal throughput). However, ATG performed adequately under all tested air traffic loads up to 1200 aircraft (225. KB/s nominal throughput). The tests show that MACS exhibits better latency performance with smaller aircraft loads when compared to ATG. The tests also show that the TestBed infrastructure successfully transmits 1200 aircraft without significant degradation of its performance. From the latency trends for both MACS and ATG, it is clear that as aircraft load increases, the latency in the system increases as well as its standard deviation.
Likewise, the trends for the update data rate for both MACS and ATG show that as the aircraft load increases, so does the standard deviation and mean of the update rates which can be attributed to the performance of MACS and ATG applications. The analysis of the results of this study have proven that the overall system performance is dependent on the individual performance of each system component that is connected to TestBed, which subsequently propagates into the system.

All TestBed characterization tests were conducted in SimLabs at NASA Ames Research Center in November 2019. This study addresses the need for a baseline TestBed characterization, and the results will serve as a reference for more complex simulation systems.