Robust Semantic Mapping and Localization on a Free-Flying Robot in Microgravity

We propose a system that uses semantic object detections to localize a microgravity free-flyer. Many applications require absolute localization in a known reference frame, such as the execution of waypoint trajectories defined by human
operators. Classical geometric methods build a map of point features, which may not be able to be associated after lighting or
environmental changes. By contrast, semantics remain invariant to changes up to the robustness of the detection algorithm
and motion of the semantic objects. In this work, we describe our approaches for both offline semantic map generation as
well as online localization against a semantic map, intended to run in real-time on the robot. We additionally demonstrate
how our semantic localizer outperforms image-feature matching in some cases, and show the robustness of the algorithm to
environmental changes. Crucially, we show in our experiments that when semantics are used to supplement point features,
localization is always improved. To our knowledge, these experiments demonstrate the first use of learned semantics for
localization on a free-flying robot in microgravity.