Is Structured Agile an Oxymoron? Tales from Implementing and Executing Agile in a US Government Environment

To paraphrase a famous quote, "No plan survives contact with the reality." Software (SW) development is often a classic example of this: whatever the plan was for a particular development, it often does not survive contact with technical realities, budget realities, program realities and schedule realities. Traditionally, SW development has followed a waterfall methodology with requirements being rigorously specified before the design, which was completed before the coding and unit testing started, which were in turn finished before validation and verification started. This model of SW engineering derives much from the HW engineering of large systems, and has been the standard methodology used in US government software acquisitions and systems for decades, with highly variable results. US Government SW requirements are built around Waterfall concepts, which assume that the plan will survive contact with reality, or at least that modifications to the plan are relatively small, and relatively few.Because of the inefficiencies and difficulties inherent in Waterfall, the commercial SW world started using a different SW development methodology called Agile more than 20 years ago. Agile believes that a plan should evolve and learn rapidly in response to the realities encountered. At its core, there are a few key elements of Agile:- A small team of people which is highly flexible and adaptive. The team collaborates and interoperates through sophisticated development architectures and release environments- An iterative, incremental development and release approach which is based upon the concept that knowledge comes from experience within the team, and that the team makes decisions based upon what it knows- A team culture which prizes transparency, inspection and adaptation. These values are necessary so that the team experience and decision making is transparent and responsive to the realities encountered during development and testingSo, how to use Agile in a US Government environment? GMSEC (Goddard Mission Services Evolution Center) develops satellite ground system software for NASA and other US Government agencies. The SW developed by the team contains a large code base of many applications used within satellite mission operations centers. It spans the full gamut of SW development types: from SW which is in a classic maintenance and sustainment mode, to new developments with a fairly well understood scope and approach, to new developments whose scope and approach are quite unclear and which require significant research and prototyping. Team members move between all of these different types of SW development. Waterfall was inadequate to the programmatic and technical needs of the team, as well as the various types of SW development being done. The software plan was not surviving contact with the technical and programmatic realities experienced by the team. To address this, the team started a small pilot project in 2016 to test the use of Agile within a small subset of the team for a new web services application. In early 2018, the use of Agile was expanded to the whole team and all the software, but we had to fulfill the NASA SW development requirements. And we needed to do this while still remaining true to the key Agile elements of transparency, inspection and adaption. In order to do this, the team worked very closely with the Software Process Improvement (SPI) team at NASA Goddard, as well as NASA engineering manageme