What Can We Learn from One Billion Ground System Log Messages?

Shortage of log-based data in a ground system they have traditionally been the under achievers in a satellite ground system. This is due to several factors: Once log messages scroll out of view on the TTC event console window they are soon forgotten. Application and system log files are scattered across directories within a system, across a multitude of servers, and across one or more databases making access cumbersome. Typical tools to perform log file content searching are generally crude and typically only employed as part of trouble-shooting exercises.As we move towards satellite constellations and fleets and add even more status information, the number of messages keeps growing. One mission now estimates that they could generate 3,000,000 messages per day 1 billion per year - for the life of their mission. What to do with those 1 billion messages? That is the challenge. With the recent technological advances in the management of large data sets, text-based processing, and data analytics, there are now capabilities that we can provide to the ground system engineers and satellite operators to address what we postulate are missed opportunities. Advanced real-time log analysis can allow us to be less reactionary in favor of being more proactive. Analytics goals include the ability to: Identify root cause of unexpected events, failures or error conditions enabled by correlating disparate data. Detect security breaches attempts before they are successful. Help admins ensure IT resources continue running optimally. Identify trends and patterns that may indicate impending failures or error conditions for valuable assets before they happen. Compare satellites in a fleet or constellation in terms of number of alarms, number of command sent to them, etc.. Answer questions like "Are the operations support needs increasing over the past year?" or "Have we seen this combination of alarm conditions before?" But really, once the tools are readily available the users will start realizing what can be done with their new powers. In this presentation we will show the results of analyzing millions of actual mission operations log messages, how the results can be displayed to the user, and how new products now available as open source can be applied to the challenges of large scale time-tagged text-based mission operations messages. Flight operations team members believe that this is a powerful new option for how they assess overall system and space asset health. Technical descriptions of the design, tools, and storage will be provided. One billion messages? Bring'em on!