Validation of a Scenario-Based Approach to Assess Gaps in Earth Observations

With the increased reliance on spaceborne Earth observation data among the Earth science community and other end users, it is important that efforts are made to promote data continuity for a range of parameters of interest. Continuity gaps may occur between missions measuring like parameters due to mission development delays or early termination and introduce the potential of increased uncertainty for retrieved parameters. To inform portfolio-level decisions for Earth observing missions, Ivanco et al. developed a method that enables the assessment of the probability of continuity gaps for multi-mission architectures and provides a framework to assess this probability in the context of multiple scenarios that represent possible future states of the architecture [Ivanco et al., “A Scenario-Based Approach to Assess Continuity Gaps in Earth Observations,” IEEE Aerospace Conf., 2024].

While this method was previously applied to assess continuity gaps for a specific multi-mission architecture, it had not yet been validated with historical data. This paper builds on the work of Ivanco et al. by utilizing data obtained from past NASA Earth science mission formulation documents to retroactively estimate mission timelines and gap probabilities for a selected multi-mission architecture. The model results are compared with the true outcomes of the missions in an attempt to validate the probabilistic gap assessment method. Discussion in this paper is limited to the probabilistic analysis portion of the method, which estimates the likelihood of gaps for a baseline mission architecture measuring a specific parameter of interest. The latter steps of the method assess the impact of decisions on measurement continuity, given the estimated gap probabilities, by use of scenario analysis and are therefore out of scope for this validation effort.

The efforts to validate the stochastic portion of the scenario-based method are not an exhaustive validation for all use cases of the method, but rather a case study of a singular architecture comparable to the architecture defined in Ivanco et al.’s analysis. The validation case study concludes that the method adequately identifies gap potential for a case where a gap was not planned in the mission architecture. A sensitivity analysis is performed to evaluate the effect of the epistemic uncertainty of input distributions on the model and demonstrates the impact of variations of input distributions related to missions in development. The validation and sensitivity analysis efforts inform future modeling improvements which will impact the effectiveness of subsequent applications of the method.

This paper first summarizes the method used to assess the probability of gaps in multi-mission architectures. It then describes the analysis performed by the authors to assess the validity of the method and quantify its predictive accuracy and presents the results of the sensitivity analysis. Lastly, insights gained from the validation efforts as well as proposed improvements pertaining to model input parameters are discussed.