A Robust Hybrid Deep Learning Model for Spatiotemporal Image Fusion

Dense time-series remote sensing data with detailed spatial information are highly desired
for the monitoring of dynamic earth systems. Due to the sensor tradeoff, most remote sensing
systems cannot provide images with both high spatial and temporal resolutions. Spatiotemporal
image fusion models provide a feasible solution to generate such a type of satellite imagery, yet
existing fusion methods are limited in predicting rapid and/or transient phenological changes.
Additionally, a systematic approach to assessing and understanding how varying levels of temporal
phenological changes affect fusion results is lacking in spatiotemporal fusion research. The objective
of this study is to develop an innovative hybrid deep learning model that can effectively and robustly
fuse the satellite imagery of various spatial and temporal resolutions. The proposed model integrates
two types of network models: super-resolution convolutional neural network (SRCNN) and long
short-term memory (LSTM). SRCNN can enhance the coarse images by restoring degraded spatial
details, while LSTM can learn and extract the temporal changing patterns from the time-series images.
To systematically assess the effects of varying levels of phenological changes, we identify image
phenological transition dates and design three temporal phenological change scenarios representing
rapid, moderate, and minimal phenological changes. The hybrid deep learning model, alongside
three benchmark fusion models, is assessed in different scenarios of phenological changes. Results
indicate the hybrid deep learning model yields significantly better results when rapid or moderate
phenological changes are present. It holds great potential in generating high-quality time-series
datasets of both high spatial and temporal resolutions, which can further benefit terrestrial system
dynamic studies. The innovative approach to understanding phenological changes’ effect will help
us better comprehend the strengths and weaknesses of current and future fusion models.